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Introduction 


During the past decade, states and districts have made significant investments in developing 
systems to measure teacher performance. The goals of such systems are to improve practice, inform 
professional growth, and ultimately improve student learning. In the very best of these systems, 
educators know the expected teaching standards and receive timely, relevant, and actionable data 
about their professional practices; districts align supports and resources for improvement; and 
educators and district staff monitor data about how instructional practices are influencing 
student learning. 


Unfortunately, not all systems work this well. Implementation often has focused on compliance and 
accountability rather than growth in teachers’ professional practice. Furthermore, the components of 
these systems (e.g., feedback/observation, professional development, and supports) often are not 
aligned with one another or do not adequately reflect or support standards-based, rigorous instruction. 
In 2016 and 2017, AIR collaborated with four districts from four states on a project to identify and 
address district-specific problems of practice (PoPs) with the current teacher evaluation and support 
systems in those districts. 


The Beyond Accountability project aimed to improve the way that districts gather and use evaluation 
system data, enabling them to better develop and support teachers, and provide them with more 
differentiated and high-quality opportunities to improve their instructional practice. To achieve this 
goal, the project employed a continuous improvement approach, specifically plan-do-study-act (PDSA) 
cycles. Each district identified a specific challenge to be addressed and conducted a root-cause 
analysis to understand the sources of the challenge. Through an iterative process, each district 
developed, tested, and revised a strategy to address the challenge. With support from AIR, each 
district collected and analyzed outcome data to inform revisions to the strategy. All but one district 
implemented multiple PDSA cycles in an approximately 18-month period during 2016 and 2017. 
(One district implemented only one cycle.) 


AIR provided technical assistance and support to within-district teams to carry out the PDSA cycle 
process, particularly in defining and reviewing data on the effectiveness of the tested strategies in 
each district. In addition, AIR brought the district teams together during the project through in-person 
and virtual meetings to share information, get feedback from one another, and learn about new or 
relevant resources or research. 


The project’s goals for each district were to improve instructional quality, increase teacher satisfaction 
with opportunities or mechanisms to improve teacher practice, and increase district capacity to engage 
in improvement efforts. Although each district’s work differed in its specific challenge area and focus, 
the project aimed to achieve the same broad outcomes across the districts. 


This report provides information about the project and the lessons learned by both AIR and the 
participating districts. It provides specific examples of participating districts’ work, including goals 
and participants, that may be useful to other districts facing similar challenges or interested in using 
similar approaches. AIR has produced a separate, comprehensive evaluation report that describes the 
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extent to which the overall project goals were achieved; the high-level findings are briefly highlighted 
in Appendix A. Appendix B provides the tools and materials used during the project (e.g., meeting 
agendas and planning templates) that other districts might use or adapt for their own work. First, 
however, we begin with an overview of improvement science in the context of a community of practice 
and why we chose this approach for Beyond Accountability. 


Continuous Improvement and Cycles of Inquiry Within 
a Community of Practice Setting 


Continuous improvement is focused research that involves multiple iterative cycles of activity for 
extended periods (Bryk, Gomez, Grunow, & LeMahieu, 2015). Schools, districts, and states have been 
engaged in continuous improvement, with varying degrees of discipline and success, for decades. 
Another name for multiple iterative cycles is inquiry processes or cycles of inquiry. Cycles of inquiry 
provide a framework for asking questions related to practice and designing rapid-cycle experiments to 
generate data that provide practitioners with information that informs that practice. Several versions of 
cycles of inquiry are used, and it is a common framework across many disciplines, including education. 


The primary purpose of employing inquiry cycles is to improve practice in applied settings. Continuous 
improvement research and cycles of inquiry differ from traditional research because they aim to better 
understand why an intervention works in some places and not others, and they involve researchers 
and practitioners in innovation design and revision (Bryk, Gomez, & Grunow, 2011; Tichnor-Wagner, 
Wachen, Cannata, & Cohen-Vogel, 2017). Employing cycles of inquiry is not the same as action 
research because action research does not address how others might learn from the findings 

and use them (Bryk et al., 2011; Tichnor-Wagner et al., 2017). 


In general, a cycle of inquiry (a) establishes current thinking; (b) identifies needs and questions; 

(c) requires some investigation of information, ideas, and data; (d) sorts the information and makes 
meaning out of it; and (e) applies the learnings. The analysis yields new action, which in turn suggests 
new inquiry into the results, so the cycle begins again (Cushman, 1999). The application of cycles of 
inquiry occurs at a variety of levels, including at the classroom level for instructional improvement and 
at the school and district levels for systemwide improvement (Park, Hironaka, Carver, & Nordstrum, 
2013). Inquiry processes can be short term or long term; however, one benefit of implementing cycles 
of inquiry is that they allow for rapidly testing changes on a small scale, testing multiple changes 
through multiple cycles, and identifying early and effective changes to action plans based on testing.* 


Informally, schools and classrooms apply cycles of inquiry on a regular basis. Formally, however, 
a few inquiry processes have emerged for use in the education field (DeLuca et al., 2015). The most 
common inquiry process is the PDSA cycle. 


1 For more information, see https://www.rand.org/content/dam/rand/pubs/monograph_reports/MR1267/mr1267.ch5.pdf. 
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Plan-Do-Study-Act Cycles 


A key feature of continuous improvement research is the PDSA cycle. These cycles test a change in 
real-world settings and help researchers determine whether a change is an improvement (Deming, 
1993; Langley et al., 2009; Tichnor-Wagner et al., 2017). During PDSA cycles, practitioners plan a 
change to be tested; do the test; study the data they collect during the test; and act on what they 
have learned from the test by abandoning, revising, or scaling up the change (Langley et al., 2009). 
Specifically, the PDSA steps are as follows: 


> Plan. Determine areas that need improvement to guide the development of an improvement 
plan. 


> Do. Implement the improvement plan by carrying out a small-scale test of the planned action. 
You can test almost any type of action, ranging from small to large. During the test, you observe 
and document any problems or unexpected events and collect data that will help you determine 
the impact of your test. 


> Study. Analyze the data you have collected and the observations you have made. Compare 
what you find with what you expected to happen and summarize what you learned from testing 
the action item. 


> Act. Use what you learned to improve your planned action. At this point, you may decide to 
either test the change again with modifications or proceed to full-scale implementation. 


When you use PDSA cycles to carry out an action plan, keep the tests as simple and straightforward 
as possible. What is most important is moving quickly through each stage of the cycle to apply what 
you learn. Figure 1 depicts the components of a PDSA cycle. 


Figure 1. PDSA Cycle Components 


ACT PLAN 
¢ Next steps: Adapt, ¢ What’s your change? 
adopt, abandon. e What's your prediction? 
e Plan to conduct test. 


STUDY DO 


© Compare results e Execute test. 
to prediction. * Collect data. 


¢ What did you learn? e Document observations. 
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Cycles of inquiry are commonly considered most effective when employed in networks, professional 


learning communities (PLCs), or other collaborative learning groups for deep, sustained learning and 
improvement at scale (Bryk et al., 2015; Cannata, Cohen-Vogel, & Sorum, 2015; Cohen-Vogel et al., 
2015; Tichnor-Wagner et al., 2017). Communities of practice are groups of people who share a 
concern, a set of problems, or a passion about a topic; they deepen their knowledge and expertise in 
this area by interacting on an ongoing basis (Wenger, McDermott, & Snyder, 2002). These communities 
serve to build the capacity of the participants to perform well, both individually and collectively, as they 


share information, build relationships, and develop effective strategies for meeting implementation 


challenges together. In short, communities of practice facilitate the creation, accumulation, and 


utilization of knowledge to improve practice among participants engaged in a shared endeavor. 


All but one Beyond Accountability district engaged in multiple PDSA cycles? related to the PoP identified 
during within-district meetings with key district and school personnel, such as directors of curriculum 


and instruction, principals, instructional coaches, and (in some cases) teachers. During the initial 


meetings, districts identified a strategy for addressing the 
PoP, developed a theory of action for how that strategy 
would address the PoP and created practical measures 
to learn whether the strategy resulted in the intended 
change. In subsequent meetings, districts analyzed data 
collected through practical measures and determined 
whether they needed to adjust the strategy. In addition 
to participating in the inquiry cycles, district teams met 
several times in-person and virtually as a community of 
practice to engage in cross-district consultations and learn 
about assessing and improving teacher instructional 
practice, continuous improvement research, and best 
practices in the field. This structure provided districts 
the opportunity to gain insight and constructive feedback 
from peers by focusing on a shared area of interest. 


The next sections include short vignettes about each 
participating district. The vignettes tell the story of each 
district’s PoP change strategies to address the PoP and 
the process for measuring their efforts and making 
improvements along the way. 


The sidebar presents key terms related to improvement 
processes that are regularly referenced throughout 
this report. 


2 One district implemented only one PDSA cycle. 


> Problem of Practice (PoP). A key issue to focus 


on for improvement. 


Root-Cause Analysis. A process through 

which participants further define and understand 
the problem that will be the focus of their efforts. 
Root-cause analysis moves participants from 
a broad problem to one that is specific enough 
on which to act. Another name for root-cause 
analysis is causal systems analysis (Bryk et al., 
2015; Proger, Bhatt, Cirks, & Gurke, 2017). 


Practical Measure. Practical measures are 
mechanisms for measuring improvements. 
They are “practical” because they can be 
quickly collected, analyzed, and used within 
the daily work lives of practitioners. Practical 
measures identify improvement targets and 
allow participants to learn continuously 
whether the changes being introduced are 
improvements (Bryk et al., 2015; Yeager, Bryk, 
Muhich, Hausman, & Morales, 2013). 
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District Vignettes 


The four districts involved in this work are from four states: (a) Kannapolis City Schools (KCS) 

in North Carolina, (b) Jackson-Madison County Schools System (JMC) in Tennessee, (c) Boone County 
Schools (BCS) in Kentucky, and (d) School District of Osceola County (Osceola) in Florida. To guide 
both the focus of the districts’ work as well as the overall project, AIR provided a driver diagram, shown 
in Figure 2. A driver diagram is a visual tool that outlines a working theory of improvement. The aim 
of the Beyond Accountability project was an evaluation system that improves teacher practice and 
instructional quality. Our hypothesis was that three primary drivers (i.e., quality data and feedback, 
teacher engagement and trust, and professional development) influence the extent to which such 
an evaluation system can be achieved. Furthermore, a host of secondary drivers sit underneath the 
primary drivers, and the districts concentrated their work in these areas. As a result, the identified 
challenges, strategies, and implementation plans varied across the four districts. 


Figure 2. Beyond Accountability Driver Diagram 


Aim Primary Drivers Secondary Drivers 


High-quality data and feedback 


on instruction Ss } Specificity of rubric rubric 


| Frequency of observations of observations 


improves teacher practice 


‘ : j Teacher engagement and trust 
and instructional quality Evaluation | Evaluation policy 


Curriculum/instructional materials 
Aligned supports and 


professional development in Coaching or peer support 


Formal professional development 


Each district identified its specific PoP using a self-assessment tool and data from teacher evaluation 
systems or related data, and the strategies to address the PoP were identified based on participants’ 
perceptions of feasibility and hypothesized impact. The self-assessment that each district completed 
to inform its PoP and related strategy can be found in Appendix B starting on page 39. The self- 
assessment is modeled after the three primary drivers and was an important step for each district’s 
stakeholders to come to consensus early in the project design about which primary driver was the 
weakest and potentially most important one in the system to address as well as which factors within 
the chosen primary driver (i.e., secondary drivers) were the most significant to consider as a focus 
for their work. The self-assessment tool helps facilitate data-driven decision making and mitigate 
preconceived notions about what the PoP or related strategies to address the PoP should be. 
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For the purposes of this section, we chose to highlight the work of only three districts—KCS, JMC, and 
Osceola. We wanted to tell the story of districts that completed at least two inquiry cycles to highlight 
the ways in which they adjusted their strategies after gathering data during a PDSA cycle. BCS began 
implementing its intervention much later than the other districts and was therefore able to complete 
only one PDSA cycle during the project. Nevertheless, the sidebar illustrates the unique and targeted 
approach that the BCS team initially designed to address its PoP 


BCS’S INITIAL CONTINUOUS IMPROVEMENT EFFORTS 
Leveraging the State’s Teacher Leadership Framework to Address Literacy Instruction 


BCS analyzed district teacher observation ratings, school instructional rounds results, student test scores, 
and data from a self-assessment of strengths and challenges in the district teacher evaluation system. 
From these data sources, BCS identified the following as its PoP: 


Despite high ratings on the Kentucky Framework for Teaching Domain 3B, Questioning and 
Discussion Techniques, other sources of qualitative data indicate that many classrooms lack 
high-level questioning, opportunities for academic discourse, and adequate use of literacy 
(oral and written) in the content areas. 


BCS indicated two possible root causes for its PoP: limited data use as part of the observation and 
feedback process and no culture of professional learning. BCS identified a single strategy to address 
its PoP and test in the first PDSA cycle: 


Instructional coaches provide professional development, through formal training and book studies, 
and model for teacher leaders high-level questioning and facilitation of student discourse applied 
to literacy. 


We know that for continuous improvement and networked improvement community initiatives to be 
successful, they must focus on or leverage goals and objectives already in place in a school, district, 
or state and integrate well into the context. In designing and implementing this strategy, BCS strategically 
tapped into the Kentucky Teacher Leadership Framework that was already embedded in the district. BCS 
selected three elementary schools and one middle school to be part of the project; two of the district’s 
instructional coaches also provided support. The theory of action for the district’s project suggested 
that after teacher leaders engaged in trainings from the coaches focused on the key instructional areas 
of interest, the teacher leaders would then be able to train other teachers in the targeted schools on these 
practices, particularly through modeling and peer observation. 
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Kannapolis City Schools 


Incorporating Peer Observations and Ongoing Administrator 
Feedback Into a Culture of Professional Development 


District Background 


KCS is a culturally diverse, midsized district located just outside Charlotte, North Carolina. As shown 
in the following tables, the district has eight schools*; employs close to 800 people, of which 385 are 
teachers; and serves approximately 5,600 students. There are six elementary schools, one middle 
school, and one high school. The district has an 85.9% graduation rate (2016-17 school year) and 
reports teacher effectiveness ratings above the state average.* 


Size 
District Number of schools Number of teachers Number of students 
Kannapolis City Schools 8 385 5,612 
Student demographics 
PENSE VA 
rati(e Hispanic/ ELA) English Low 
tad Islander Black Latino American White Other learners income 
panhapell 1.5% 289% 31.0% 0.3% 331% 5.6% 9.1% 78.1% 
City Schools 


In 2015, KCS released its 5-year strategic plan, which outlined that the district seeks improvements 
in, among other things, curriculum, instruction, and student services. The strategic plan also says that 
the district aims to increase student performance to meet all federal and state targets and exceed 
expected student academic growth by 2020. Similarly, the district aims to increase its graduation rate 
to 90% by the 2019-20 academic year. To improve overall curriculum and instruction, KCS seeks to 
align professional development to support the goals of the strategic plan.° 


w 


In addition to the district’s eight schools that serve K-12, KCS also has one Head Start/Child Development Center that serves 
3- and 4-year-olds. 


8 


See http://www.kannapolis.k12.nc.us/news/archived_news/k_c_s_graduation_rate_hits_record_high_in_2017 and http://www. 
ncpublicschools.org/docs/intern-research/reports/teachereff2014. pdf. 


a 


See http://www.kcs.k12.nc.us/news/archived_news/k_c_s_adopts_new_five-year_strategic_plan/. 
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Problem of Practice 


The current teacher evaluation system in KCS has been in place since 2009. Instruments and rubrics 
used for evaluation are based on the North Carolina Professional Teaching Standards and the Framework 
for 21st Century Learning (from the Partnership for 21st Century Skills). KCS uses observations, self- 
assessments, student surveys, artifacts, Educator Value-Added Assessment System (EVAAS) data, 
and district walk-through instruments to measure instructional practice. Ideally, teachers and principals 
use the data from these instruments to inform their professional learning plans and request support. 
Furthermore, principals should use observation and EVAAS data about teacher practice to identify 
teacher leaders, assign mentors, and make staffing decisions. The district uses the walk-through data 
to determine professional development needs and offerings. KCS wanted to revisit this process as it 
played out in practice as part of the Beyond Accountability project. 


The KCS Beyond Accountability team consisted of a core leadership team (the district’s assistant 
superintendent, the K-12 curriculum and professional development coordinator, the director of 
secondary and student services, and the director of elementary and Title | services). At times, a 
larger team of stakeholders (three principals, two assistant principals, three instructional coaches, 
one media coordinator, and one teacher) also participated. During the first within-district meeting that 
included both the core leadership team and the larger stakeholder group, four data sources illustrated 
issues in the evaluation system and helped identify the PoP to be addressed as a part of the project: 


> Self-assessment of system strengths and challenges (see Appendix B, p. 39) 


> Teacher Working Conditions Survey (2015-16): professional development and beginning 
teacher data 


> North Carolina Educator Evaluator System: Summative teacher evaluation 


> Instructional walk-through data 


After analyzing all the data sources, the KCS core leadership team identified some of the challenge 
areas on which to focus its work. Specifically, the team focused on the fact that the data consistently 
revealed that the system was not conducive to teachers learning from one another to increase self- 
reflection and improve teacher practice. For example, in the Teacher Working Conditions Survey 
(2015-16), beginning teachers reported a lack of observations of and by mentors and peers. In 
addition, teachers indicated that schoollevel follow-up from facilitators and administrators was lacking, 
and feedback often was not communicated. As they analyzed these data points, the KCS team voiced 
difficulty in choosing “one problem” to address because some seemed contingent on or interrelated 
with others. In collaboration with KCS, AIR used Handout 4 on page 52, “Develop Potential Problems 
of Practice,” to help the district hone in on a PoP they considered important, worthy of buy-in to address, 
and feasible. Ultimately, the KCS team felt that peer observation opportunities and a process to support 
professional learning were the appropriate problems to address through this process. As such, the 
district embraced the opportunity to use a continuous improvement process to create a structure for 
peer observation and test its effects on instruction and collaboration. The KCS team summarized 
the problem as follows: 


Teachers do not have sufficient opportunities to learn from each other. 
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Addressing the Problem of Practice Through the Continuous 
Improvement Process 


To address the PoP that teachers do not have sufficient opportunities to improve instruction 
by learning from one another, KCS identified two main root causes: 


> Peer learning is not prioritized. 


> An existing formal structure to support peer observation is absent. 


The district selected the following as the overarching strategy to address the PoP 


Create a peer observation structure to support teachers learning from one another's instruction. 


Plan-Do-Study-Act Cycle 1 


Rather than immediately incorporate the new peer observation process with the formal evaluation 
process, the district chose to proceed with a phased implementation to test the observation process 
with a small number of staff in a few schools before expanding and formalizing the process with staff 
from all schools. KCS limited its work to a group of four teachers from four schools in Grades 3-9. 
The teachers took part in four peer observations guided by one district facilitator. The curriculum and 
professional development coordinator assumed the role of the observation facilitator during Cycle 1. 


The coordinator and other members of the district core leadership team developed a three-part peer 
observation process (preobservation conference, the observation, and a postobservation conference) 
that would be applicable in any content area. To support the process, the district created a protocol 
that identified the purpose of peer observations and the learner-centered observation model, which 
set expectations and established the roles of those participating in the observation. 


The three main participants in each observation were the observing teacher, the teacher being observed, 
and the observation facilitator. The observing teacher determined the instructional focus for the 
observation and took ownership of the learning process. The teacher being observed provided 
background, modeled the designated instruction, and participated in the reflective postobservation 
process. The observation facilitator guided the teachers though the three-stage process and 
coordinated follow-up support for the observing teacher. Participants conducted observations 
during the observing teacher’s free periods and were provided with coverage and release time during 
observation periods. 


Prior to the preobservation conference, the observing teacher and the observation facilitator defined 
a focus for the observation based on the observing teacher’s needs. Participants were advised that 
the observation focus should be supported by needs evident in the teacher’s self-assessment, the 
teacher’s professional development plan (PDP) goals, and prior observations; it was to be aligned 
with school and district improvement goals. The observation focus was posed as a question through 
which the observer could collect data and identify strategies to apply in his or her own classroom. 
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During the preobservation conference, the observation facilitator met with the observing teacher 
and teacher being observed to 


> clarify the focus for the observation, 

> schedule the observation, 

> share information about the specific lesson to be observed, 

> determine the note-taking method for the observation, 

> discuss other relevant observation details (e.g., etiquette, resources), 

> schedule the postobservation meeting time, and 

> complete the preobservation meeting form. 
As mentioned above, during the scheduled observation time, the observing teacher followed the 
agreed-on etiquette and recorded notes relevant to the instructional focus area/PoP As soon as 
possible after the observation, the observation facilitator met with the observing teacher and 


teacher being observed for a postobservation conference and systematically guided reflection 
on the observation according to the following steps: 


> Review the main area of focus/PoR 


> Have the observing teacher share highlights from his or her notes and ask any clarifying 
questions to the teacher being observed. 


> Have both the observing teacher and the teacher being observed reflect on the lesson. 
> Identify strategies that the observing teacher can apply and have the observing teacher 


set next steps. 


The observation facilitator’s role was to complete a preobservation meeting form hosted on Google 
Docs during the preobservation conference. This form enabled the facilitator to track each observation 
and make brief notes on the focus and logistics of each observation. The preobservation form asked 
the observing teacher and the teacher being observed to answer questions such as the following: 


> What is the focus area and focus question of the observation? 

> When will the observation take place? 

> Has the observing teacher decided on a note-taking method? 

> Has the teacher being observed shared details about the students, classroom, and lesson? 
> When do we plan to have the postobservation meeting? 

> What substitute coverage is needed? 


> Other comments or questions? 
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In addition, KCS created a postobservation form that prompted observing teachers and teachers being 
observed to review the observation and establish goals for after the observation. The postobservation 
section asked both individuals to review the following: 


> What are the next steps for the observing teacher? 


> What follow-up support is needed, and who will provide the support? 


To gauge the effectiveness of these observations and whether they were perceived to improve 
instruction and peer collaboration, the district, with support from AIR, developed a set of surveys as 
practical measures. The district administered a survey to its initial group of teachers to measure their 
perception of the value of the peer observation process, the extent to which observing teachers felt 
the support was tailored to their professional development needs, and the ways in which these 
observations improved their knowledge as it related to their professional development goals. A 
survey administered to the observation facilitators measured their perception of the value of the peer 
observation process to the observing teacher and the teacher being observed. The survey assessed 
the value of the peer observation process to facilitators themselves and gauged their confidence in 
guiding the peer observation process. Further, the pre- and postobservation notes were used as 
practical measures and gave the district further insight into the practicality of the process and 
whether observing teachers were applying the skills learned from the observation. Ultimately, the 
district’s understanding of the data collected by way of the practical measures led them to expand 
the use of creating a peer observation structure to support teachers learning from others’ instruction 
and move to a second PDSA cycle. For example, in the Cycle 1 reflection surveys, all four teacher 
observers agreed or strongly agreed that their knowledge was increased through the process, particularly 
in areas of professional development needs (which was how their observation session was intended to 
be aligned). Furthermore, all observing teachers in the first cycle identified an immediate and relevant 
next step for classroom application. Last, the reflection surveys also suggested that the observation 
facilitators universally agreed that there was value in the process for the teacher observing and the 
teacher being observed. 


Plan-Do-Study-Act Cycle 2 


After the first PDSA cycle, KCS realized that the process of applying what was learned during peer 
observations needed to be more active. The postobservation tool was a good start; however, the 
district wanted to make sure that teachers followed through on applying their knowledge in a speedy 
and deliberate manner. Because it aimed to test the strategy with a wider range of teachers throughout 
the district in Cycle 2, the team worked to refine the peer observation process. The feedback generated 
from Cycle 1 led to the development of a Teacher Instructional Log for Cycle 2 that required teachers, 
within 2-3 weeks of their observation, to record and identify their applications of professional learning 
to their instruction and their students’ reactions to new or changed instructional strategies. 
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Before initiating Cycle 2, the district opened the process to all teachers. One hundred twenty-nine 
teachers expressed interest in the peer observation process by way of a survey, and 68 teachers 
were selected to participate. The district expanded its peer observation structure and held as many 
as 27 observations led by 11 observation facilitators. The district also expanded the format of these 
observations and offered observing teachers and teachers being observed the opportunity to participate 
in individual, paired, small-group, or large-group observations. This led to 86 teachers ultimately 
impacted by the new observation process. In addition, 10 more observation facilitators were trained. 


Teachers’ experiences with the observation process in Cycles 1 and 2 were borne out in the internal 
evaluation data as well. AIR gathered teacher survey data for the pilot and comparison schools in 
KCS from fall 2016 through spring 2017. Respondents from both comparison and pilot schools 
reported a double-digit increase between fall 2016 and spring 2017 in the extent to which they said 
that the school or district made available opportunities to observe expert teachers modeling skills that 
were related to their feedback. In that same time period, teachers from the pilot schools reported 

a double-digit increase in the extent to which they pursued those opportunities or found those 
opportunities useful. 


To determine Cycle 2’s impact, KCS used all the measurement tools from Cycle 1 and added the 
Teacher Instructional Log to document the ways in which observing teachers applied the knowledge 
and skills gained from peer observations in their own classrooms. The results from practical measures 
data collected during Cycle 2 showed positive feedback from the participants—an overwhelming 
majority saw the process as valuable (see Figure 3 for Teacher Reflection Survey data). Most observing 
teachers felt that the process increased their skills in facilitating student learning and increased their 
knowledge of content or instructional strategy (see Figure 4 for Observation Facilitator Survey data). 
From the Teacher Instructional Log, 68% of the participants said that they applied their learning 
from the observations. The remaining 32% expressed their intention to do so next year. 


However, as Cycle 2 progressed, the district faced a new set of challenges. The timing of the work, with 
most of it taking place toward the end of the school year, limited teachers’ ability to put skills learned 
from the peer observations into practice. The district team also recognized that large-group observations 
did not necessarily meet teachers’ individual needs. In addition, despite positive feedback, not all 
participants recognized the connection between their observations and their larger PDP goals. As the 
district concluded Cycle 2, it acknowledged an emerging need to strengthen alignment of the district 
evaluation system and teacher PDPs with the peer observation process. KCS determined that the 
focus of the peer observation had to be driven more intentionally by teachers’ PDP goals or areas 
of improvement that emerged from administrator evaluations. To do so, the district also would need 
to schedule peer observations as they fit into the evaluation system, with ample time before the end 
of the school year for teachers to apply their learning in their classrooms. 
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Figure 3. Practical Measures From the Teacher Reflection Survey Data From Cycle 2 


The support | received while participating in the Participating in the peer observation process 
peer observation process was tailored to my increased my knowledge with respect to subject 
professional development needs. area content and/or instructional strategies in 


my professional development goal(s). 


1% 
13% 
55% 
32% 
Participating in the peer observation process Participating in the peer observation process 
increased my skills to facilitate student learning is valuable. 


as related to my professional development goal(s). 


2% 5% 


11% 
43% 
44% 


@ Completely @ 104 great extent @ 10 some extent @ Notatall 
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Figure 4. Practical Measures From the Observation Facilitator Survey Data From Cycle 2 


The peer observation process is valuable for 
peer observers. 


The peer observation process is valuable for 
observation facilitators. 


The peer observation process is valuable for the 
teachers being observed. 


14% 


e 


| feel confident in my ability to facilitate the peer 
observation process. 


43% 


@ Completely @ 104 great extent @ To some extent 


Plan-Do-Study-Act Cycle 3 


The district team decided that the refined peer observation process would be adopted and embedded 
into the teacher evaluation process as an elective, targeted professional development strategy. Still, 
teacher observations conducted by school administrators needed to be explicitly tied to teachers’ 
PDP goals to create a stronger connection among goals, observation and feedback, and professional 
development support. For school administrators to better make those connections, administrators 
needed to more regularly observe classroom instruction and provide specific feedback to teachers to 
improve practice and suggest appropriate professional development, which could include but is not 
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limited to peer observations. Therefore, for Cycle 3, which was ongoing as this report was being 
developed, the district hopes that administrators will 


> learn to make providing instructional feedback to staff a priority in their schedule, 
> schedule and complete weekly observations and feedback meetings, 

> use the tracking system consistently and with fidelity, 

> feel confident in giving targeted feedback on PDP goals, 

> increase the amount of feedback they provide teachers, and 


> feel supported by their weekly central office support meetings. 


The district hopes that teachers will 


> feel confident that administrators could identify a specific area of feedback that would 
improve instruction and 


> feel supported in their PDP goals by a more regular observation and feedback process. 


Because of its work, KCS hopes to implement ongoing improvements to educator quality policies and 
practices across the district. This will allow the district to align teachers’ PDP goals with feedback 
from observations that ultimately support professional learning. KCS continues to develop practical 
measures, including an Observation and Feedback Tracker for principals to chart increased 
observation and feedback potentially related to teacher instructional goals. Overall, KCS hopes 
to strengthen alignment of the district evaluation system and teacher professional development. 


Working to Sustain Improvements 


By conducting the continuous improvement work, the phased implementation approach allowed the 
district to measure and adjust the scope of its strategy to create a peer observation process so that 
teachers could learn from one another’s instruction. The gradual and planned involvement of more 
teachers between Cycles 1 and 2 allowed the district to scale its initial work. The district’s pivot to 
focusing on the formal evaluation process in Cycle 3 was informed by the need to have administrators 
provide more frequent and targeted observation and feedback to teachers and align the feedback with 
PDP goals and the needs and opportunities for professional development. It is possible that this was 
always the more high-leverage approach to what the district was seeking to achieve related to improving 
instruction and student learning; but, for a variety of reasons, they did not prioritize this early in the 
project as the PoP or the strategy to address the problem and instead chose to focus initially on 
building a culture and set of routines for peer learning. As the work progresses, the district would 

like to consider how to enhance and support administrators’ observation and feedback processes 
even further by creating “look-for” documents aligned to key areas of improvement in the district 

or other tools and rubrics to support targeted, standards-aligned observation and feedback. These 
resources also may be useful for the established peer observation process. 


Various district- and school-level KCS stakeholders, including the district’s assistant superintendent 
and key central office leaders, representative principals, assistant principals, instructional coaches, 
a media coordinator, and a teacher, were involved in developing, implementing, extending, and supporting 
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this strategy. By engaging key stakeholders in designing a peer observation process and a tracking 
system with embedded data collection tools and then moving forward on improving feedback to teachers, 
KCS has started to develop an infrastructure more likely to sustain the desired change of supporting 

a culture of teachers as a community of learners. From its work, the district reported that it has 
implemented a more strategic and deliberate approach to improvement. PDSA cycles provided the 
district with a model to clearly identify and quantify problems and allowed the district to anchor 
themselves in data before planning next steps. The district also reported that through this work, 
administrators are motivated to prioritize inquiry cycles and improvement science as they have seen 
teachers use feedback to improve their instruction, and they recognize that through this work, staff 
are more connected to school needs and successes. 


Jackson-Madison County School System 


Developing Administrators’ Understanding of Content Standards to 
Better Provide Teachers Explicit, Actionable, and Aligned Feedback 


District Background 


JMC is in a small city in an urbanized area in Madison County, Tennessee.® As shown in the following 
tables, the district has 27 schools, employs close to 900 teachers, and serves 12,500 students. In 
the 2015-16 school year, the district had a 92.4% graduation rate, which was higher than the state 
average of 85.5%. Almost 72% of the district’s students come from low-income households. 


Size 


District Number of schools Number of teachers Number of students 


Jackson-Madison 


2t 7 172: 
County School System 89 500 


Student demographics 


Asian/ 


Pacific Hispanic/ Native English Low 
Ea Islander ETH 4 Latino American White Other learners income 


Jackson-Madison 
County School System 


1.4% 58.4% 7.0% n/a 30.9% 2.3% 4.0% 71.8% 
The current teacher evaluation system in JMC has been in place since 2011. Several rubrics are used 
for teacher evaluations in Tennessee. However, JMC uses the Tennessee Educator Acceleration Model 
(TEAM), a model based on the Danielson and Marzano instructional frameworks. Principals use data 
from this evaluation tool to identify potential teacher leaders, learning coaches, and mentors. In 
addition, these data measure overall teacher effectiveness, and specific indicators drive professional 


® Data are from the National Center for Education Statistics (https://nces.ed.gov/ccd/districtsearch). 
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development and training. Principals also make recommendations about data relative to tenure, 
personnel, and differentiated pay. 


Problem of Practice 


Prior to joining the Beyond Accountability project, JMC had already identified early literacy as a key 
district priority for improving student learning. The district participates in the Leading Innovation for 
Tennessee Education’ network, which is a group of Tennessee districts focusing together on improving 
instruction and professional learning, particularly early literacy. The district had analyzed instructional 
rounds data that illustrated a discrepancy between the expected rigors of instruction versus what was 
observed in practice. These data revealed that most of the lessons illustrating this gap focused on 
developing students’ foundational reading skills. Most phonics, phonological awareness, and other 
foundational skills instruction was explicit, and teachers were clearly working to build students’ reading 
and decoding skills by the end of third grade. However, in many cases, the lessons did not yet fully 
align to grade-level standards, and it was not clear that students were systematically building the skills 
they needed to learn to read from grade to grade.’ The district also had teacher perception survey data 
that provided insight regarding teachers’ perceptions of the principal’s role in facilitating professional 
learning and growth opportunities as well as in educator development and support.° Considering all 

of this information, JMC identified the following as its focal PoP: 


Administrators lack understanding of the academic shifts required to effectively teach the 
standards, which prevents them from providing teachers with feedback and coaching that is 
explicit, actionable, and aligned with the demands for college and career readiness. 


Addressing the Problem of Practice Through the Continuous 
Improvement Process 


JMC’s Beyond Accountability team consisted of a core leadership team (the district superintendent, 
the chief academic officer, an academic coordinator, and a school principal) plus a larger team of 
stakeholders (two data coaches, the district’s academic literacy coordinator, the leader of assessment 
and accountability, the leader of professional development, and two additional school principals) to 
implement the work. The district team selected the following as the overarching strategy to address 
the PoP 


Provide training to administrators to help them develop greater knowledge of instructional shifts 
in the appropriate content standards so that they may be better able to determine if instruction 
accurately addresses the demand of the standards and are better able to provide teachers with 
explicit and actionable feedback and coaching that is aligned with the demands of college and 
Career readiness. 


’ See https://lifteducationtn.com/wp-content/uploads/2017/11/LIFT-Education-Annual-Report-201 7_FINAL. pdf. 


8 Adapted from the Jackson-Madison Instructional Review Summary. 


° From the JMC Teacher Perception Survey of spring 2016. 
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Plan-Do-Study-Act Cycle 1 


To address its PoP the JMC team developed a training strategy to test in the first inquiry cycle that 
aimed to grow administrators’ knowledge of instructional shifts in grade-appropriate standards. 
The training focused on the following areas: 


> On- and off-stage evidence 

> Rubric connections (standards-aligned conversation with high-quality feedback) 
> Feedback that specifically addresses shifts and the rigor of the content 

> Observing standards in action and what standards-aligned instruction looks like 


> Practicing with frequent accountability pieces for ensuring implementation, otherwise known 
as the bridge to practice 


The district collaborated with AIR to design practical measures to track the outcomes of its intervention. 
AIR facilitated the creation of a survey instrument that gauged administrator perceptions of English 
language arts (ELA) instructional shifts and their ability to determine if the rigor of the standards was 
met in instruction. Other items on the survey helped gauge the relevance of district-led trainings on 
the topic, how valuable administrators found the trainings, and the confidence level and preparedness 
of administrators in understanding shifts in the ELA standards. 


The JMC team initially piloted the work and planned to later expand it to more schools across the 
district. The pilot began with administrators in six elementary schools. All principals and assistant 
principals received the training at their regularly scheduled monthly meetings, and the practical 
measure gathered information from administrators in the six pilot schools. 


JMC conducted a “before and after” survey of administrators in the six pilot schools: the before version 
was administered 1 week before the training and the after version a few days after the training. The 
findings between both surveys were similar, suggesting that administrators may have overestimated their 
knowledge and understanding of the instructional shifts in the survey administered before the training. 
Therefore, the district team concluded a need to break down the process and address smaller parts of 
the rubric at a time to delve deeper into each one during the training. Furthermore, to provide more 
accurate and specific data to inform decision making in the next PDSA cycle, AIR and JMC revised 

the practical measure and reconsidered the process for data gathering to attain more accurate data. 


Plan-Do-Study-Act Cycle 2 


Building on prior knowledge and leveraging existing work within the district, the team developed an 
ELA classroom observation tool based on a tool they had used for another initiative as well as on the 
Instructional Practice Guide: Coaching Tool from Student Achievement Partners. The ELA classroom 
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observation tool, designed to facilitate administrators’ ability to identify rigorous instruction and provide 
better, more instructionally specific feedback and coaching, outlines three general core actions: 


> Focus each lesson on a high-quality text (or multiple texts). 


> Employ questions and tasks—both oral and written—that are text specific and accurately 
address the analytical thinking required by the grade-level-standards. 


> Provide all students with opportunities to engage in the work of the lesson. 


After concluding that the training needed to be broken into smaller parts, the JMC team refined 
the Cycle 2 plan to focus on a narrower scope and concentrate on only one of these core actions: 


Develop administrators’ capacity to identify the alignment with high-quality text(s) and grade- 
level standards in preparation for feedback and coaching that is explicit and actionable through 
instructional rounds, discussions on the data collected, modeling coaching conversations, and 
practicing coaching conversations. 


For the second inquiry cycle, instructional coaches and district team members partnered with 
administrators in the six pilot schools to provide more targeted support for the focused core action 

in Cycle 2. Each pair (school administrator and instructional coach/district team member) conducted 
“norming observations” of four ELA classrooms to gather baseline data and calibrate observations. 
Both the instructional coach/district team member and the school administrator gathered data using 
the ELA classroom observation tool. Together, they compared and discussed the collected data. The 
instructional coach/district team member modeled a coaching conversation based on the gathered 
data, and the school administrator practiced conducting a similar coaching conversation. Each pair 
conducted a second round of classroom observations using the same observation tool as in the 
norming observations. 


This new approach allowed the leadership team to provide training that specifically focused on one 
core action of the ELA classroom observation tool. The data from the ELA classroom observation 
tool served as the practical measure for Cycle 2 and helped the district gather data that were more 
accurate of administrators’ understanding of ELA instructional shifts rather than administrators’ 
perceptions of their own understanding and ability to provide feedback. 


Comparisons of the data gathered in the norming observations versus data gathered in the second 
round of observations revealed that as administrators gained familiarity with the tool, their observations 
were more aligned with those of the instructional coach and the district team members, and they 
improved their ability to identify high-quality texts and instruction using the tool. The team concluded 
that this approach facilitated the gathering of more accurate data and allowed them to better identify 
specific areas in which school administrators needed additional training. 
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Plan-Do-Study-Act Cycle 3 


The data gathered in Cycle 2 helped the JMC team identify even more specific areas of need for 
administrator training. In Cycle 3, the team decided to focus on training administrators on additional 
core action areas of the observation tool. Consequently, the team developed the following strategy: 


Prepare administrators for feedback that is explicit, actionable, and aligned to the demands of 
the standards and train on Core Actions 1B, 1C, and 1D.*° 


Core Action 1B: The text(s) are at or above the quantitative and qualitative complexity level 
expected for the grade and time in the school year. 


Core Action 1C: The text(s) exhibit exceptional craft and thought and/or provide useful 
information to build knowledge. 


Core Action 1D: Students connect the acquisition of skills to making meaning from reading. 


The observation tool was revised to include space for observers to begin to identify areas of focus 
for the coaching conversation and notes for coaching feedback. 


In the third inquiry cycle, the district team and instructional coaches provided whole-group administrator 
training on the additional core action areas. For this cycle, administrators and coaches used videos 
from prior exercises to conduct these trainings because teachers were on summer vacation.** Once 
again, instructional coaches and district team members partnered with administrators from the six 
pilot schools to conduct ELA classroom observations using the revised observation tool. This was 
followed by whole-group administrator training on identifying an individual teacher's next steps that are 
explicit, actionable, and aligned to the full demands of the standards. A second round of observations 
using the revised observation tool followed the training. Each administrator’s ratings and notes in 
preparation for a coaching conversation were compared with ratings and notes of the partnered 
instructional coach/district team member. 


The gathered data revealed greater alignment between administrators’ observations and those of 
the instructional coaches/district team member for the additional core action areas. However, the 
team decided that the observation tool needed further revision to allow observers to cite evidence 
for their ratings during the instructional rounds. The extent to which teachers perceive this alignment 
in their experiences with the observation and feedback process as well as the influence this growth 
in administrators’ knowledge and capacity has on teaching and learning remains to be seen and is 
something the district should continue to monitor. 


10 The language for these Core Actions was adapted from Student Achievement Partners’ ELA/Literacy Coaching Tools for Grades K-2 
and Grades 3-12. See https://achievethecore.org/category/1155/printable-versions. 


11 At the time of the third inquiry cycle, the school year had ended, and in-person classroom observations were not possible. Consequently, 
the JMC team partnered with the National Board of Professional Teaching Standards to use the ATLAS library of authentic videos of 
National Board Certified Teachers delivering instruction as the “classroom” to be observed. 
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Working to Sustain Improvements 


Through their continuous improvement work, the JMC team developed its understanding and capacity 
to engage in the continuous improvement process while also developing trainings and a practical, 
rigorous, and focused classroom observation tool that more school administrators throughout the 
district could use. The team began with broad and ambitious goals for administrator training but 
quickly realized a need to significantly scale back the content of its training. Although the overarching 
goal of the training did not change, the incremental steps toward that goal became much more 
focused to gradually implement smaller pieces of the training, gather data with a continually 
improved observation tool, and use that data to inform next steps. 


School District of Osceola County 
Doubling Down on Standards-Based Planning and Instruction 


District Background 


Osceola is a culturally diverse, large district located just outside Orlando, Florida. As shown in the 
following tables, the district has 52 schools, employs 3,655 teachers, and serves 61,000 students. 
The current superintendent, Dr. Debra Pace, began her tenure in February 2016, having served in the 
district for 25 years and most recently as the associate superintendent of human resources in 
neighboring Brevard County. She brings to the district a mind-set of “using research and data to 
guide decision making.”*? 


Size 


District Number of schools Number of teachers Number of students 


School District of Osceola County 52 3,655 61,000 


Student demographics 


PNET A 


Pacific Hispanic/ Native English Low 
(ad Islander ETH 4 Latino American White Other learners income 


School District of 


2% 11% 59% <1% 25% 3% 19% 63% 
Osceola County 


For its 2016-2019 strategic plan, Osceola identified goals related to academic success, talent 
management, fiscal responsibility, community engagement, and governance. For academic success, 
Osceola hopes to increase the high school graduation rate from 81% to 86% by the 2018-19 


12 See https://today.ucf.edu/educational-leadership-doctoral-grad-appointed-as-new-school-district-of-osceola-county-superintendent/. 
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academic year. To do so, the district established multiple strategies, including ensuring high levels of 
learning for all students in literacy and mathematics by cultivating teachers’ deep understanding of 
content and instruction in these areas. 


Problem of Practice 


Osceola uses the Marzano Teacher Evaluation Model to evaluate teacher performance.*® The district 
gathered data on the occurrence of various elements of the Marzano Learning Map from formal and 
informal observations conducted in classrooms across grade levels. When tallied, the district team 
noticed that elements from the observation rubric** most commonly observed were those related to 
helping students interact, practice, or deepen new knowledge. Those conducting the walk-throughs, 
including instructional coaches and other district leadership, observed very few instances that focused 
on rubric elements related to helping students generate and test hypotheses. Elements of instruction 
focused on cognitively complex tasks occurred half as often as those for practicing skills. Therefore, 
the district team decided to tackle the following PoP: 


Teachers too frequently focus on students’ practicing skills rather than on the more important 
work of engaging students in cognitively complex tasks. 


Addressing the Problem of Practice Through the Continuous 
Improvement Process 


The Osceola core leadership team consisted of the district’s director of secondary curriculum and 
instruction—middle school education and the principal of one of the district’s elementary schools. 
A larger team of stakeholders, including district instructional framework coaches and school-based 
coaches, supported implementation of the work. 


The team identified the root cause of its PoP as a lack of understanding the intent of the standards 
and how they connect to the instructional framework, particularly during the planning process. The 
team decided that if teachers better understood the application of the instructional framework and 
had improved skills to plan instruction that involved cognitively complex tasks to engage students, 
district data on this instructional strategy would improve. Based on the identified root cause, the 
district selected the following as the overarching strategy to address the PoP: 


Implement common collaborative planning time with additional structures and resources, 
including use of the Marzano Teaching Map, modeling of effective planning by instructional 
leaders and coaches, and guidelines for effective instructional plans (such as guiding questions, 
self-assessments, samples, and a description of plan components for execution aligned with the 
standards and instructional framework). 


48 See http://www.marzanoevaluation.com for more information. 


4 See http://www.marzanocenter.com/files/LearningMap_AST_Framework_Evaluator_20120226.pdf. 
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The district posited that providing additional guidance and support for high-quality instructional 
planning would lead to increased teacher understanding of the standards, the instructional framework, 
and their connection and, therefore, improved teaching and learning. The team used Handout 6 starting 
on page 56 of Appendix B, particularly the questions related to “likelinood for success,” “feasibility,” 
and “scalability” to hone in on common collaborative planning time as the vehicle by which to guide 
and support high-quality instructional planning. For example, they decided to leverage PLCs that 
were already operating in schools across the district. 


Plan-Do-Study-Act Cycle 1 


For its first inquiry cycle, Osceola leveraged existing teacher common collaborative planning times to 
focus on the use of the Marzano Teaching Map, model effective planning by instructional leaders and 
coaches, and provide guidelines for effective instructional plans. Osceola established the following 
action steps to implement its first cycle: 


1. District instructional framework and curriculum coaches (IFCs) will develop guidelines 
for effective instructional plans. 


2. IFCs will share guidelines with school curriculum coaches and staff. 
3. IFCs will obtain feedback to revise and finalize the guidelines. 


4. School coaches will schedule professional development to model the guidelines, teaching 
maps, and Florida standards in three 2-hour sessions. 


5. School-based coaches and leaders, as well as IFCs, will check for implementation using 
walk-throughs, the reflective visit form, and instructional rounds. 


Osceola recruited 44 K-5 teachers at one elementary school for the first cycle: eight teachers each for 
Grades K-3 and six teachers each for Grades 4 and 5. This school was selected because it had been 
on academic probation but had a strong, improvement-oriented principal. District IFCs were tasked with 
creating instructional planning guidelines for the teachers. These guidelines included identifying the 
cognitive complexity of the standard or depth of knowledge level, the academic vocabulary and skills 
needed by students, the critical content-learning targets, the strategies used to deliver the learning 
targets, and the success criteria. 


The IFCs then provided school-based coaches with training to model the new guidelines. The two 
school-based coaches from the elementary school then worked to model use of the teaching map 
for standards-based planning to a PLC lead so they could then, in turn, model it for teachers during 
regularly scheduled PLC time. If that was not possible, the school-based coach attended PLC meetings 
to model the use of the planning guidelines. The district conducted three professional development 
sessions in fall 2016. The district, with AIR’s support, created short surveys to assess educators’ 
perceptions of the effectiveness of guidance and support provided by the district for instructional 
planning and linking standards to the instructional framework. The teacher survey asked teachers 
about their knowledge of Florida’s standards, the teaching strategies described in the Marzano 
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instructional framework, the connection between the two, and the support they received for standards- 
based instructional planning. The district also evaluated teachers’ ability to engage in high-quality 
planning by scoring teachers’ instructional plans using a rubric reflective of the instructional planning 
guidelines developed by the IFCs. 


The team also created observational protocols for coaches and principals to use during walk-throughs 
and instructional rounds to document data, reflect on their observations, and look specifically for 
evidence of improved alignment. Coaches and principals relied on the school reflective visit protocol 
to check for ongoing application of trainings and professional development. 


Overall, the district learned that many teachers found the professional development to be helpful. 
However, two challenges emerged: (a) There was inconsistent teacher buy-in and enthusiasm for the 
trainings, and (b) some teachers did not fully understand how the training directly connected to the 
classroom and lesson plans. Based on the team’s review of the data, the district decided to focus 

on helping teachers understand the purpose of the trainings and how the work should be embedded 

in day-to-day activities. The results from the instructional lesson planning rubric, developed by the 
district IFCs, also revealed little use by teachers of the guiding questions for standards-based planning. 
Therefore, the district decided to adapt its first strategy, with adjustments, to help teachers intentionally 
plan by using the guiding questions and other specific strategies from the instructional framework. 


Plan-Do-Study-Act Cycle 2 


For Cycle 2, the district narrowed the number of teachers (now only K-2 teachers) involved in the 
project and worked to provide support in understanding specific instructional strategies, planning for 
the intentional use of these instructional strategies, and implementing the instructional strategies. 
IFCs developed professional development for teachers related to “recording and representing 
knowledge” and “examining reasoning” strategies as well as modeled lessons based on those 
strategies. To implement Cycle 2, the district completed two tasks: 


> Developed materials and additional plans that included model lessons on two instructional 
strategies—recording and representing knowledge plus examining reasoning—with respect 
to one essential standard in mathematics and ELA for each of Grades K-2. 


> Delivered a full-day professional development session for K-2 teachers that included the 
recently developed model lessons. 
After these trainings, teachers were now required to do the following: 


> Deliver a lesson on the same instructional strategies that had been modeled during 
professional development. 


> Turn in samples of student work. 


Osceola and AIR conducted short pre-post surveys to assess the application of what participants 
had learned during the professional development session. Figure 5 shows the results of the surveys. 
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Figure 5. Osceola PDSA Cycle 2 Teacher Professional Development Pre-Post Survey Results 


| understand how to connect the standards and the teaching strategies in the Marzano instructional framework to design 


my instructional plans and materials. 


55% BB Strongly agree 
BB Agree 


BH Somewhat agree 


2% 


BB Disagree 


12% 


5% 


Time 1 Time 2 


Aggregate results from the pre-post surveys suggest improvements in teachers’ perception of their 
ability to connect the standards and teaching strategies focused on “recording and representing 
knowledge” and “examining reasoning” to better design instructional plans and materials. Not shown in 
Figure 5 are disaggregated results that suggest participants had slightly less understanding of how to 
make these connections for “examining reasoning” than for “recording and representing knowledge.” 


To assess the extent to which the survey results were evident in student learning, teachers for each 
grade level brought in student work that directly pertained to the standards and strategies on which 
they had focused. A team of raters scored the student work using a common rubric. There were lessons 
learned about how to design and implement this practical measure, particularly for standardizing the 
selection of student work to rate across the group of teachers as well as better calibrating ratings. 
Nevertheless, analyzing student work alongside the pre-post survey results suggested to the district 
that the standards-based instructional strategies, particularly examining reasoning, were only partially 

in place in the classroom, and some teachers still were not making the connection between the 
professional development and their lessons. 


Cycle 2 strengthened the district’s understanding about how to continue to refine training and 
professional development, how to provide teachers with more exposure to a variety of instructional 
strategies, and how to refine ways to embed standards-based lesson planning into the school process. 
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Plan-Do-Study-Act Cycle 3 


For Cycle 3, the district expanded the training it provided to a larger group of teachers across the 
district, including adding additional grades. The district also discussed the continued importance of 
having district IFCs, elementary curriculum folks, and school-based coaches at the table. In fall 2017, 
the district offered additional professional development, with an emphasis on the instructional strategies 
identified in the Marzano Teaching Map, which was scheduled during collaborative planning times. 
These strategies included a deeper exploration and understanding of “examining reasoning” as well 
as adding “elaborating on new information” and “analyzing similarities and differences.” AIR and the 
district were revising the measures to study Cycle 3 based on learnings from Cycle 2 at the time of 
this report’s development yet continued to focus on a combination of professional development, 
model lessons, and student work. 


Working to Sustain Improvements 


The Osceola team focused on a challenge area that was aligned to its district strategic plan and goals, 
and the Beyond Accountability project allowed the district to accelerate its learnings and progress on 
those goals. The efforts of the Osceola team during this project set the district on a path for sustained 
change related to the way data are collected about teacher performance, professional development to 
support standards-based lesson planning, and ongoing continuous improvement efforts at the school 
and district levels. 


Through this initiative, the district adjusted the walk-through process and now has a cadre of IFCs 
conducting more frequent classroom walk-throughs based on the Marzano Teaching Map and the 
Marzano Learning Map. The district is then able to use the formative walk-through data to identify, on 
a regular basis, key areas of need related to teacher practice. The district provided teachers with time 
and professional development so that they can better understand how to effectively implement the 
instructional framework for standards-based planning and instruction. Teachers also benefited from 
having the implementation of these strategies modeled by IFCs within their classrooms. Furthermore, 
the work conducted by Osceola demonstrates the importance of a bridge to practice for implementation 
in the classroom; the district now has an improved structure for professional development that includes 
formal sessions plus model lessons and an examination of student work. The overall professional 
development “package” aimed to help teachers understand instructional strategies, plan for using these 
instructional strategies, and implement the instructional strategies. 


Osceola is now looking forward to adapting the continuous improvement process in other schools within 
the district. Two elementary, one middle, and one high school have been selected as part of the next 
phase of the rollout. Each school will explore a different PoP based on a similar districtwide issue. 
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Getting Beyond Accountability: What Did We Learn? 
What Did the Districts Learn? 


AIR supported four districts in conducting intensive, disciplined continuous improvement cycles that 
aimed to improve practice and supports for teachers as well as build capacity to apply this level of 
continuous improvement going forward. AIR and the district participants learned a great deal about 
this process; here we list some of the most salient lessons for others who want to engage in similar 
efforts to consider. 


Leadership Commitment Is Crucial 


School and district leadership must be on board and commit to a common set of goals and outcomes 
to engage in successful, disciplined continuous improvement. Much of the work related to continuous 
improvement will happen in classrooms and at the school level, so staff must feel confident that 
district leadership (a) continuously supports their time and effort, (b) will collaborate with them 

to inform the process, (c) invest in the success of the initiative where relevant, and (d) showcase 
how the work contributes to the district’s overall goals and strategic plans. District leadership in the 
Beyond Accountability districts was essential in the following ways: 


> Share performance data to guide the work. 
> Create the time and space for the work to take place. 


> Communicate about the initiative in a positive and common way across multiple schools 
and staff. 


> Tap district resources, such as instructional coaches, where necessary. 


> Take the lessons-learned and evidence of improvements seriously and consider pathways 
for broader implementation. 


> Reaffirm the importance of getting better at district-level systems designed to improve 
instruction and learning as well as use inquiry cycles as a mechanism to do that. 


Buy-in Across All Participants Is Essential, So Is Creating a Culture 
of Trust and Collaboration 


Even though school and district staff have been engaging in continuous improvement efforts for a 
long time, doing it in the way that the Beyond Accountability project required was new for most of the 
participants. Districts need to ensure that communication about the work is clear and consistent and 
that participants understand not only the mechanics of the work but also the goals and their role in 
the effort. In addition, district administrators and project leaders need to reinforce the notion that 
this work goes beyond evaluation and demonstrate how these projects work to improve overall 
teaching and learning processes. Commitment and trust permits true partnerships, engagement, 
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and collaborative practices. Administrators should provide points of contact to the project for 
participants to weigh in and ask questions about the work. Furthermore, administrators must be 
willing to adapt their cycles as teachers provide feedback and as teachers’ needs evolve and develop. 


The Role of Data and High-Quality Measurement Cannot Be 
Underestimated, and Support for Data Collection, Analysis, and 
Interpretation Must Be Provided 


Data and measurement play an important role in two distinct parts of the continuous improvement 
cycle. First, AIR encouraged the districts to use multiple sources of data, such as walk-through data, 
evaluation data, and teacher perception survey data, to best understand the PoP and, in some cases, 
the root cause of the problem. One district commented that the root-cause analysis process was 
essential to targeting its strategies. These data were shared in the early stages of each district’s 
project and were the driving force behind conversations among district team participants about what 
they should focus on for improvement and why. 


In addition, AIR supported the districts in developing and analyzing practical measures data for each 
cycle of the process, particularly during transitions from one cycle to the next and homing in on 
specific areas of a strategy or intervention that needed adjustment or recalibration. In many cases, 
this was the most challenging component of the work—identifying the right practical measure (both 
the content and the method) that will illustrate improvements (or not) is essential but tricky work. The 
district teams’ understanding of and commitment to using these practical measures helped promote 
their strategies. Project participants from across the districts often noted that school and district staff 
rarely if ever took the time to measure these types of short-term outcomes and instead frequently 
“guessed” or “felt” how an initiative was going. In some cases, districts took a very active role in 
developing and using the practical measures, particularly identifying outcomes based on their theory 
of action, making predictions for each outcome, and collectively making sense of data to produce 
results for each outcome. 


Put Parameters on the Focus of the Project, but, Ultimately, Schools 
and Districts Must Concentrate on What They Can Realistically Achieve 


Beyond Accountability set out to address challenges with feedback and observation systems and the 
way that evaluation data were used to improve instructional practice and access professional supports. 
For a variety of reasons, most districts chose to identify where the data showed weaknesses in specific 
areas of instructional practice (using evaluation data as well as other sources of data) and implemented 
professional development or supports to address those areas. These districts channeled their focus 
on mechanisms for improving instructional practice outside the formal evaluation process and, in 
some cases, later considered how the findings might influence the teacher evaluation and support 
system. Concentrating the work in this way allowed district and school staff to build capacity for 

the continuous improvement process and achieve early wins in a low-risk environment. 
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Allow the Theory of Action to Lead the Way and Adjust It as You Learn 


The districts were required to develop a theory of action to articulate the focus of each inquiry cycle. 
Designing and adhering to a meaningful theory of action, with short- and long-term outcomes, that has 
the potential for real change can be challenging. Several key questions should be considered when 
developing a theory of action, including the following: 


> Why are we prioritizing this strategy or intervention? Does it respond to the PoP we identified? 


> What, specifically, are we asking all the actors involved in the theory of action to do or do 
differently? The implementation of the strategy(ies) in a high-quality and targeted way is the 
key to moving from “if” we do this, “then” this will be our outcome. We cannot shortchange 
the implementation aspect of a theory of action. 


> How will the strategy work to achieve the outcomes we expect to see? 


e AIR took a mostly hands-off approach to the design and implementation of the strategy 
itself; that work was primarily the responsibility of the schools and districts. 


e Other organizations supporting continuous improvement work may want to consider taking 
a more involved role in the design and implementation of the strategy. 


> What makes us think that if we implement the strategy as proposed, it will ultimately achieve 
the outcome (e.g., improve teacher practice and student learning)? Relatedly, identify what 
success looks like at the outset. If the overall goal is to improve some aspect of teacher 
practice, what exactly does that look like? What are the measures? 


Facilitating PDSA Cycles Is Intensive Work 


Each Beyond Accountability district completed between one and three PDSA cycles in slightly more 
than 1 year with the support of school and district staff as well as at least one AIR dedicated facilitator. 
The plan, study, and act phases require at least daylong conversations on the front and back ends of 
doing the implementation work for a few months. Participants mentioned that doing disciplined PDSA 
cycles in this way was sometimes uncomfortable because it is very specific and concentrates on 
outcomes and effectiveness in a way that can be difficult in the everyday life of school and district 
staff. They often do not have the protected time to work through specific questions using such a 
process but now have a structure for sustaining the approach. Nevertheless, key questions remain 
for the schools and districts about how they will sustain this work, including exactly who, how, and 
when they will embed PDSA cycles in school and district work. 


Timing and Coherence Are Everything 


We quickly learned that this work must be part of a larger strategic plan and set of goals in the school 
or district or it will not succeed. School and district staff are simply too busy to have this level of 
intensive work put on their plate unless it supports them in achieving larger goals and outcomes that 
they are already working toward. Participants noted that starting with small focus areas allowed them 
to determine how best to embed the work into the larger system and scale to others in the school 
and/or district. 
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Because classrooms are situated in and inextricably linked to the broader school system, and consistent 
with research, we found that teachers are better able to sustain change when mechanisms are in 
place at multiple levels of the system to support their efforts. These evidence-based mechanisms 
include the following: 


> The presence of a supportive professional community of colleagues in the school that reinforces 
normative changes and provides continuing opportunities to learn (McLaughlin & Mitra, 2001; 
Payzant, 2005; Stokes, Sato, McLaughlin, & Talbert, 1997) 


> Knowledgeable and supportive school leadership (Anderson-Butcher, Lawson, Bean, Boone, & 
Kwiatkowski, 2004; Berends, Bodilly, & Kirby, 2002; Comer, Haynes, Joyner, & Ben-Avie, 1996; 
Datnow, Hubbard, & Mehan, 2002; Fullan & Stiegelbauer, 1990; Hargreaves & Fink, 2000; 
McLaughlin & Mitra, 2001; McPartland, Balfanz, Jordan, & Legters, 2002; Muncey & McQuillan, 
1996; Murphy & Datnow, 2003) 


> Connections with other schools or teachers engaged in similar reform (Cooper, Slavin, & Madden, 
1998; McDonald et al., 1999; Muncey & McQuillan, 1996) 


> Normative coherence or alignment between the district policy context and the reform (Berends 
et al., 2002; Comer et al., 1996; Datnow et al., 2002; McLaughlin & Mitra, 2001) 


Networks, in All Their Different Forms, Should Be a Fundamental 
Approach to Continuous Improvement 


The cycles and strategies developed by each district varied in scope and design. As indicated by the 
vignettes, the focus of each district’s work was on a range of secondary drivers of improvement— 
changes that might take place at the district level rather than changes that might happen across 
districts. This limited, to some extent, the group of districts from operating like a true networked 
improvement community,*® which often benefits from the coordination of PoPs, solutions, and data 
collection practices. Nevertheless, the districts came together as a community during the in-person 
and virtual cross-district convenings. This networking and interdistrict sharing was essential for learning 
about different approaches and getting low-stakes peer feedback on the content of the work as well 
as the approach to the PDSA cycles. Participants routinely referenced through meeting surveys and 
anecdotally that the cross-district consultancy was a valuable process they engaged in at the cross- 
district meetings to check assumptions, get peer feedback, and refine their approach. 


+8 AIR defines a networked improvement community as a group of stakeholders from diverse backgrounds solving problems 
together through a cycle of Plan-Do-Study-Act (https://www.air.org/resource/using-networked-improvement-communities-improve- 
educational-practice). 
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Disciplined Continuous Improvement Work Provides Space to Improve 
on Content and Process 


Supported continuous improvement has two chief benefits: the opportunity to learn about what kinds 
of strategies or interventions work in which contexts and at which levels (school, district, state) and 
the opportunity to grow the capacity of participants to apply disciplined inquiry cycles to a myriad of 
challenges or PoPs that emerge in their work. AIR asked key district participants to complete a “scale 
and sustainability” questionnaire to help each district think through next steps for continuing and 
spreading the work of the Beyond Accountability project. Selected responses from that questionnaire 
included the following: 


4 4 There is power in the process of truly reflecting on what your areas of need/ 
problems are and then working through the PDSA cycle. As a school, we have grown 
tremendously because of it. y | Y 


4 4 The continuous model is important for us to identify a specific problem and work 
as a team as we try to solve the problem with a research-based approach. The data 
focus allows participants to constantly reflect on the effectiveness of this plan. y | Y 


44 In addition to the instructional value from peer/mini observations, | believe 
the process has helped improve our understanding of implementation science and 
equipped us to better plan and implement initiatives in the district. y | Y 


44 Self-reflection is necessary to improve. Using the Plan-Do-Study-Act model 
requires reflection but also accountability in your work to make the process become 
action and action become improvement. y | Y 
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Appendix A. 
Summary of Internal Evaluation Findings 


AIR conducted an internal evaluation to determine whether AIR and its partner districts met the project 
goals. AIR developed a teacher survey to assess progress toward meeting two of the project goals: 
improve instructional quality and boost teacher satisfaction with opportunities to improve their practice. 
The design of the evaluation did not allow for the direct measurement of instructional quality; however, 
through the teacher survey, the research team was able to ask questions about conditions for 
instructional quality improvement, such as teachers’ motivation to change. The following sections 
report findings from the teacher survey designed to answer two summative evaluation questions: 


1. To what extent are conditions right for improvements in instructional quality? For example, 
are teachers motivated to change? To what extent do teachers believe that the effort required 
to change their practice is worth it? 


2. To what extent are teachers satisfied with opportunities or mechanisms to improve their 
practice? For example, what feedback do teachers receive, and do they understand and 
agree with the feedback? Do professional development opportunities exist that are aligned 
with the feedback that has been received? 


To answer these questions, the survey asked teachers about their teaching experience, beliefs about 
teacher evaluation systems, feedback received from observers, and opportunities for professional 
growth. AIR administered the teacher survey to pilot and comparison group teachers in three of the 
four participating districts.*° Pilot group teachers participated in elements of the districts’ projects 
aimed at improving the use of evaluation and instructional practice data and improving teacher 
development and support systems, whereas comparison group teachers continued with typical 
school and district business. This appendix provides a high-level overview of the key findings from 
the survey. The full set of findings is available in the cross-district survey report. 


Teaching Experience 


At the fall administration of the survey, pilot group teachers were slightly less experienced than 
comparison group teachers. They had an average of 11 years of teaching experience, whereas 
comparison group teachers had an average of 14 years of teaching experience. Pilot group teachers 
also had slightly less experience teaching in their districts and schools. They had an average of 8 years 
in their district and 5 years in their schools, whereas comparison group teachers had an average of 
10 years in their district and 7 years in their schools. The pilot group teachers who participated in the 
spring administration of the survey had nearly identical teaching experience as the pilot group teachers 
who participated in the fall administration. The same was true of the comparison group teachers. 


16 Responses from BCS teachers are not included in this cross-district summary report because AIR staff members administered 
the spring survey to pilot group teachers only. 
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Conditions for Instructional Improvement 


Overall, AIR staff members found evidence of improvements in the conditions for instructional quality 
improvement with respect to general perceptions of the evaluation system, perceptions of evaluators’ 
knowledge and skills, and perceptions of feedback received. For example, the percentage of pilot 
group teachers who agreed or strongly agreed that the evaluation system does a good job distinguishing 
among teachers of different performance levels increased by 10 percentage points between 2016 
and 2017, yet there was no increase for comparison group teachers. In addition, the percentage of 
pilot group teachers who agreed or strongly agreed that being observed or receiving feedback increases 
their level of stress or anxiety declined by 5 percentage points between 2016 and 2017, yet there was 
a decline of less than 2 percentage points for comparison group teachers. 


Teachers’ Satisfaction With Opportunities to Improve Their Practice 


AIR asked teachers about three types of opportunities to improve their practice: meeting with 
instructional leaders who can support them as they implement suggestions provided in their 
feedback, observing expert teachers modeling skills related to their feedback, and planning for 
implementing new strategies based on their feedback. AIR found some evidence that pilot group 
teachers’ satisfaction with three types of opportunities to improve their practice increased across 
time relative to the satisfaction of comparison group teachers. For example, between 2016 and 2017, 
the percentage of pilot group teachers and comparison group teachers who reported that opportunities 
to meet with an instructional leader were moderately or highly useful increased by 5 percentage 
points and 2 percentage points, respectively. In the same period, the percentage of pilot group 
teachers who rated opportunities to observe expert teachers as moderately or highly useful increased 
by 5 percentage points, whereas a slight decrease occurred for comparison group teachers. Finally, 
in the same period, the percentage of pilot group teachers who rated opportunities to plan for 
implementing new strategies based on observers’ feedback as moderately or highly useful increased 
by 3 percentage points, but a slight decrease occurred for comparison group teachers. 
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Appendix B. 
Project Tools and Guidance 


This appendix provides an overview of the tools and resources used to support districts in the Beyond 
Accountability project. The materials included here are somewhat generic, although the project team 
customized the materials for each cycle of inquiry to specifically meet the needs of each district 
throughout the project. Handouts 1 and 2 were used only once. The activities in Handouts 3-8 were 
used, with small adjustments, for each cycle of inquiry. A short description of each handout follows. 
The actual handouts follow the descriptions. 


HANDOUT 1. Self-Assessment 


District teams may enter the project with preconceived notions of the district’s PoP regarding the 
district’s teacher evaluation and support system. This self-assessment is a good resource to help 
reach consensus regarding existing challenges (or PoPs) and focuses the conversation on one of three 
common challenges: (a) measurement/data collection and reporting/information quality, (b) the use 
of data and supports, and (c) teacher trust and engagement. 


What you will need: 


> One printed copy of the handout for each participant 


> One copy of each rating table, preferably on a large poster, where the team will record the 
district consensus 


> Approximately 2 hours 


After this activity, you may want to use Handout 4 to help the team develop PoP statements and 
prioritize the key challenges. 


HANDOUT 2. Plan-Do-Study-Act Sample 


Handout 2 will help participants understand the PDSA process and help them see how each cycle 
builds on knowledge gained from the previous cycle. The sample in the handout also should help 
participants see how each cycle aims to address only a small piece that informs gradual steps toward 
the bigger goal of creating the continuous improvement process. 


Ideally, you should first review the PDSA process and then provide the handout as an example. Discuss 
what the example shows about the PDSA cycle. In the discussion, make sure to point out that cycles 
occur rather quickly. Small changes are made, data are quickly and easily gathered, and these data 
inform the next steps. The overarching goal may be ambitious. However, the changes made in each 
cycle should be small, attainable, and easily measurable. 

What you will need: 


> One printed copy of the handout for each participant 


> Approximately 30 minutes 
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HANDOUT 3. Review Data 


Handout 3 provides guidance for reviewing a set of data about the district/schools that are the focus 
of the project. This initial data piece should be existing data to help identify the PoP The presentation 
of the data may require work, ahead of the initial review, to ensure that the data are presented ina 
way that is easy for participants to review and interpret. 


What you will need: 
> One printed copy of the handout for each participant 
> Printed copies of the data for participants to review 


> Approximately 30 minutes 


HANDOUT 4. Develop Potential Problems of Practice 


In this activity, all participants will have an opportunity to share what they believe to be the biggest 
area of difficulty based on the data reviewed in the previous activity. The group will begin with a broad 
list of problem statements and gradually home in on one PoP to be addressed through this work. This 
activity can take time, but it will allow for small- and large-group discussion, ultimately arriving at one 
agreed-on PoP 


Hint: The clearer the data reviewed in the previous activity, the easier and faster it will be for the team 
to agree on a PoP The facilitator will need to guide the group to identify a clear and specific PoP A very 
broad problem statement will make it difficult to identify the root cause of the problem and, later, a 
strategy to address it. 


What you will need: 


> One printed copy of the handout for each participant 


> Chart paper to draft and finalize the agreed-on PoP (You will likely need to revise the statement 
several times before reaching a consensus. It may be easier to type this using a projector and 
screen for the whole group to see and contribute to refining the statement.) 


> Approximately 60 to 90 minutes 


HANDOUT 5. Identifying Root Causes 


This activity will help further refine the specific cause of the PoP that will be tackled through this 
work. Acknowledging that there might be multiple causes for the problem, the handout provides rating 
criteria to help identify a root cause that is of most importance, is more likely to be perceived as a key 
issue and receive support from others (stakeholders), and is most feasible. 
What you will need: 

> One printed copy of the handout for each participant 

> Sticky notes 


> Approximately 30 minutes 


Copyright © 2018 American Institutes for Research. This handout may be reproduced for district/school use in planning and 
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HANDOUT 6. Determine a Strategy and Develop a Theory of Action 


This activity guides the group though the process of brainstorming possible strategies to address the 
root cause of the PoP and then prioritize one strategy to address through this work. The activity 
concludes with the development of a theory of action. This activity is a whole-group activity. 
What you will need: 

> One printed copy of the handout for each participant 

> Chart paper or digital (computer, projector, and screen) copy of the table in Step 2 

> Chart paper or digital equipment to jointly develop the theory of action 


> Approximately 60 minutes 


HANDOUT 7. Create a Plan for Implementation 


This tool provides district teams with a template that they can use to outline plans for implementation, 
including who will carry out which aspects of the strategy, where and when the strategy will be carried 
out, what supports or resources will be needed, and what implementation challenges are anticipated. 
The tool also provides a set of important discussion questions regarding implementation that district 
teams should review and discuss together before completing the implementation plan template. 

What you will need: 


> One printed copy of the handout for each participant 


> Digital copy of the template so that participants can type into the template to develop their 
own plan(s) 


> Approximately 60 to 90 minutes 


HANDOUT 8. Measures and Predictions 


Handout 8 provides a brief overview of practical measures and some examples to help participants 
understand what practical measures are and begin to think about what measures they can use to 
determine progress in their own work. It includes a table to outline expected outcomes, measures, 
and data to collect to determine progress toward those expected outcomes plus predictions of what 
the collected data will likely review (the predictions should align with the expected outcomes). 


What you will need: 
> One printed copy of the handout for each participant 
> Digital copy of the table so that participants can log their information 


> Approximately 30 minutes (plus time for team members to draft and finalize the practical 
measure for gathering data) 
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HANDOUT 1. | Self-Assessment 


Individually respond to the following self-assessment statements and answer the questions that 
follow. Then discuss with your district team and record your district’s consensus responses in the 
repeated tables that start on page 46 of this appendix. 


1. Measurement/ Data Collection and Reporting/Information Quality 


Strongly Strongly 


disagree Disagree Agree agree 


la. Teachers, principals, and district staff have an agreed-on 
definition of high-quality instructional practice that can be 
measured through observations, student surveys, or other means. 


1b. Teachers, principals, and district staff receive information about 
teacher practice often enough to help them consider changes 
that could support improvements in practice. 


1c. Teachers, principals, and district staff receive accurate 
information about teacher practice. 


1d. Teachers, principals, and district staff find the information they 
receive about teacher practice detailed or specific enough to be 
useful and actionable for supporting improvements in practice. 


le. Teachers, principals, and district staff have easy and ready access 
to information about teacher practice. 


1f. Measured instructional practices relate to student learning. 
Staff believe that strong performance is linked to student growth 
and learning. 


1g. Measures of teacher practice show meaningful differences 
in the quality of instruction for individual teachers across time 
and for different teachers. 


Looking at the statements that you rated as disagree or strongly disagree, answer the following questions: 


1. Why did you disagree? Describe what you see as the challenge and give an example that 
illustrates it. For example, if you disagreed that educators get accurate data about their 
practice, explain specifically what you mean. Is there little interrater reliability, such that 
teachers get inconsistent information about their practice? Is it another issue? 
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2. Is this a systemic challenge (i.e., does it affect the instructional practice of many 
teachers)? Would most others (teachers, principals, or other district staff) agree that 
this is a systemic challenge? 


3. What data or evidence do you have to demonstrate that this is a systemic challenge? 
Would most others agree that this is a systemic challenge if presented with the data or 
evidence? If you do not have data or evidence now, what would you need to collect and 
how could you do it? Data or evidence might include results from surveys; interviews; focus 
groups with teachers, principals, or district staff; the analysis of data about teacher practice 
or student learning; or findings from personal observations or conversations with staff. 


Looking at the statements that you rated as agree or strongly agree, answer the following questions: 


4. Why did you agree? Describe the situation and give an example that illustrates it. 
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5. Would most others (teachers, principals, or other district staff) agree with this statement? 


6. What data or evidence do you have to support this statement? Would most others agree 
with the statement if presented with the data or evidence? If you do not have data or 


evidence now, what would you need to collect and how could you do it? 


2. Use of Data and Supports 


Strongly 


disagree Disagree Agree 


Strongly 
agree 


2a. Teachers have access to sufficient supports linked to identified 
needs in their practice (e.g., coaching, professional development, 
or peer mentoring). 


2b. Teachers have formal opportunities to build on recognized 
strengths of their practice (e.g., peer mentoring or coaching). 


2c. Teachers use information about instructional practice to make 
changes in their practice. 


2d. Principals use information about instructional practice to align 


resources and supports that can improve instructional practice 
and student learning (e.g., identify potential teacher mentors, 
make classroom assignment decisions, assign coaches or 
peer mentors, and make recommendations about tenure or 
other personnel decisions). 


2e. District staff use information about instructional practice to 
align resources and supports that can improve instructional 
practice and student learning (e.g., determine professional 
development needs/ offerings for the district and evaluate 
professional development or instructional materials). 
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Looking at the statements that you rated as disagree or strongly disagree, answer the following questions: 


7. Why did you disagree? Describe what you see as the challenge and give an example that 
illustrates it. For example, if you disagreed that teachers use information about instructional 
practice to make changes in their practice, explain specifically what you mean. Is it because 
they do not have the right information to use? They do not know how to integrate the 


information into their practice? Something else? 


8. Is this a systemic challenge (i.e., does it affect the instructional practice of many 
teachers)? Would most others (teachers, principals, or other district staff) agree that 
this is a systemic challenge? 


9. What data or evidence do you have to demonstrate that this is a systemic challenge? 
Would most others agree that this is a systemic challenge if presented with the data or 
evidence? If you do not have data or evidence now, what would you need to collect and 
how could you do it? Data or evidence might include results from surveys; interviews; focus 
groups with teachers, principals, or district staff; the analysis of data about teacher practice 
or student learning; or findings from personal observations or conversations with staff. 
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Looking at the statements that you rated as agree or strongly agree, answer the following questions: 


10. Why did you agree? Describe the situation and give an example that illustrates it. 


11. Would most others (teachers, principals, or other district staff) agree with this statement? 


12. What data or evidence do you have to support this statement? Would most others agree 
with the statement if presented with the data or evidence? If you do not have data or 
evidence now, what would you need to collect and how could you do it? 
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3. Teacher Trust and Engagement 


Strongly Strongly 


disagree Disagree Agree agree 


3a. The goals and rationale for the district's evaluation system are 
clear and well documented. 


3b. All staff understand the components of the evaluation system, 
the process of measurement, and how it is to be carried out. 
Systematic documentation and training are in place. 


3c. Teachers believe the evaluation system is intended to help 
improve instructional practice, and they welcome measurement 
of their practice. 


3d. Principals and other district staff believe the evaluation system 
is designed to help improve instructional practice. 


3e. Teachers, principals, and district staff can articulate how measures 
of teacher practice are used to improve practice within the district. 


3f. Teachers are involved on an ongoing basis in providing input 
about the evaluation system measures, implementation, and 
use of information. 


Looking at the statements that you rated as disagree or strongly disagree, answer the following questions: 


13. Why did you disagree? Describe what you see as the challenge and give an example that 
illustrates it. For example, if you disagreed that the goals and rationale for the district’s 
evaluation system are clear and well documented, explain specifically what you mean. 


14. Is this a systemic challenge (i.e., does it affect the instructional practice of many 
teachers)? Would most others (teachers, principals, or other district staff) agree that 
this is a systemic challenge? 
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15. What data or evidence do you have to demonstrate that this is a systemic challenge? 
Would most others agree that this is a systemic challenge if presented with the data or 
evidence? If you do not have data or evidence now, what would you need to collect and 
how could you do it? Data or evidence might include results from surveys; interviews; focus 
groups with teachers, principals, or district staff; the analysis of data about teacher practice 
or student learning; or findings from personal observations or conversations with staff. 


Looking at the statements that you rated as agree or strongly agree, answer the following questions: 


16. Why did you agree? Describe the situation and give an example that illustrates it. 


17. Would most others (teachers, principals, or other district staff) agree with this statement? 


18. What data or evidence do you have to support this statement? Would most others agree 
with the statement if presented with the data or evidence? If you do not have data or 
evidence now, what would you need to collect and how could you do it? 
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District Consensus 
1. Measurement/ Data Collection and Reporting/Information Quality 


Strongly Strongly 


disagree Disagree Agree agree 


la. Teachers, principals, and district staff have an agreed-on 
definition of high-quality instructional practice that can be 
measured through observations, student surveys, or other means. 


1b. Teachers, principals, and district staff receive information about 
teacher practice often enough to help them consider changes 
that could support improvements in practice. 


1c. Teachers, principals, and district staff receive accurate 
information about teacher practice. 


1d. Teachers, principals, and district staff find the information they 
receive about teacher practice detailed or specific enough to be 
useful and actionable for supporting improvements in practice. 


le. Teachers, principals, and district staff have easy and ready access 
to information about teacher practice. 


1f. Measured instructional practices relate to student learning. 
Staff believe that strong performance is linked to student growth 
and learning. 


1g. Measures of teacher practice show meaningful differences 
in the quality of instruction for individual teachers across time 
and for different teachers. 


2. Use of Data and Supports 


Strongly Strongly 


disagree Disagree Agree agree 


2a. Teachers have access to sufficient supports linked to identified 
needs in their practice (e.g., coaching, professional development, 
or peer mentoring). 


2b. Teachers have formal opportunities to build on recognized 
strengths of their practice (e.g., peer mentoring or coaching). 


2c. Teachers use information about instructional practice to make 
changes in their practice. 


2d. Principals use information about instructional practice to align 
resources and supports that can improve instructional practice 
and student learning (e.g., identify potential teacher mentors, 
make classroom assignment decisions, assign coaches or 
peer mentors, and make recommendations about tenure or 
other personnel decisions). 
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Strongly Strongly 


disagree Disagree Agree agree 


2e. District staff use information about instructional practice to 
align resources and supports that can improve instructional 
practice and student learning (e.g., determine professional 
development needs/ offerings for the district and evaluate 
professional development or instructional materials). 


3. Teacher Trust and Engagement 


Strongly Strongly 


disagree Disagree Agree agree 


3a. The goals and rationale for the district's evaluation system are 
clear and well documented. 


3b. All staff understand the components of the evaluation system, 
the process of measurement, and how it is to be carried out. 
Systematic documentation and training are in place. 


3c. Teachers believe the evaluation system is intended to help 
improve instructional practice, and they welcome measurement 
of their practice. 


3d. Principals and other district staff believe the evaluation system 
is designed to help improve instructional practice. 


3e. Teachers, principals, and district staff can articulate how measures 
of teacher practice are used to improve practice within the district. 


3f. Teachers are involved on an ongoing basis in providing input 
about the evaluation system measures, implementation, and 
use of information. 
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HANDOUT 2. | Plan-Do-Study-Act Sample 


This handout is adapted with permission from the Agency for Healthcare Research and Quality (AHRQ 
Publication No. 15-0023-EF; February 2015). 


TOOL: Patient Feedback 


STEP: Dissemination of Surveys 


Cycle: First Try 


PLAN 

| plan to test a process of giving out satisfaction surveys and getting them filled out and back to us. 
| hope this produces at least 25 completed surveys per week during the campaign. The steps to 
execute are as follows: 


> We will display the surveys at the checkout desk. 


> The checkout attendant will encourage each patient to fill out a survey and put it into the box 
next to the surveys. 


> We will try this for 1 week. 
DO 
What did you observe? 


> We noticed that patients often had other things to attend to during this time, such as making 
an appointment or paying for services, and did not feel they could take on another task during 
this time. 


> The checkout area can get busy and backed up at times. 
> The checkout attendant often forgot to ask patients to fill out a survey. 
STUDY 
What did you learn? Did you meet your measurement goal? 
> We had only eight surveys returned at the end of the week. This process did not work well. 
ACT 
What did you conclude from this cycle? 


> Patients did not want to stay to fill out the survey once their visit was over. We need to give 
patients a way to fill out the survey when they have time. 


> We will encourage them to fill it out when they get home and offer a stamped envelope to 
mail the survey back to us. 
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Cycle: Second Try 


PLAN 

| plan to test a process of giving out satisfaction surveys and getting them filled out and back to us. 
| hope this produces at least 25 completed surveys per week during the campaign. The steps to 
execute are as follows: 


> We will display the surveys at the checkout desk. 


> The checkout attendant will encourage each patient to take a survey and an envelope. 
Patients will be asked to fill out the survey at home and mail it back to us. 


> We will try this for 2 weeks. 
DO 
What did you observe? 


> The checkout attendant successfully worked the request of the survey into the checkout 
procedure. 


> We noticed that patients also had other papers to manage during this time. 

> Only about 30% of the customers per checkout attendant took a survey and an envelope. 
STUDY 
What did you learn? Did you meet your measurement goal? 

> We had only three surveys returned at the end of 2 weeks. This process did not work well. 
ACT 
What did you conclude from this cycle? 


> Some patients did not want to be bothered at this point in the visit; they were more interested 
in checking out and leaving. 


> Once patients step out of the building, they will likely not remember to do the survey. 


> We need to approach them at a different point in their visit when they are still with us— 
perhaps when they are waiting for the doctor and have nothing to do. 


Cycle: Third Try 


PLAN 

| plan to test a process of giving out satisfaction surveys and getting them filled out and back to us. 
| hope this produces at least 25 completed surveys per week during the campaign. The steps to 
execute are as follows: 


> We will leave the surveys in each examination room next to a survey box with pens and pencils. 


> We will ask the nurses to point the surveys out or hand them out after vitals and suggest that 
while patients are waiting they could fill out the survey and put it in box. 


> We will see after 1 week how many surveys we collected. 
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DO 
What did you observe? 


> Upon self-report, most nurses indicated they were good with pointing out or handing the survey 
to the patient. 


> Some patients may need help reading the survey, but the nurses are too busy to help. 


> On a few occasions, the doctor came in while the patient was filling out survey, so the survey 
was not complete. 


STUDY 
What did you learn? Did you meet your measurement goal? 
> We had 24 surveys completed at the end of 1 week. This process worked better. 
> We need to figure out how to assist people who may need help reading the survey. 
ACT 
What did you conclude from this cycle? 
> Approaching patients while they are still in the clinic was more successful. 


> Most patients had time while waiting for the doctor to fill out the survey. 
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Handout 3. | Review Data 


Using the guiding questions that follow, record two or three findings in the chart (or use sticky notes, 
if directed by your facilitator). Findings are short statements of information, written clearly. They are 
quantitative or qualitative facts from the data that are meaningful to you. 


GUIDING QUESTIONS 


1. What were the highs, lows, and averages seen in the data? What data were about what you 
expected? What data were above or below what you expected? Are there any other surprises 
in the data? 


2. What patterns or trends did you notice or would you expect in the data across time? Did the 
data or finding change or do you expect it to change? 


3. What is your main finding? Circle it in your list. 


FINDINGS 


Example: Thirty-six percent of the teachers disagreed or strongly disagreed that feedback 
from their evaluator was evidence based. 


Example: The highest proportion of teachers with less than “Effective” ratings was in the 
“questioning” techniques element. 


Example: Twenty-two percent of the teachers reported that scheduled observations were good 
examples of their everyday instruction. 
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HANDOUT 4. | Develop Potential Problems of Practice 


Step 1 


Individually or in small groups, based on the major findings from your district’s data, develop a potential 
PoP to address in this project. (If working with a group of 10 or more, work in smaller groups to help 
quickly identify one PoP that the whole group agrees on.) 


Examples 


> Many principals struggle to provide teachers with useful, authentic evaluation feedback that 
reflects a concrete understanding of effective instructional practice. 


> Districts do not effectively use information from the teacher evaluation process to design and 
implement professional development and ongoing support for teachers. 


> The highest proportion of teachers with less than “Effective” ratings was in the questioning 
techniques element. 


Write the PoP in the following table and be ready to explain in what ways addressing it will improve the 
system beyond accountability. The problem(s) you identify must fit within one of the three “big ideas” 
for this project (i.e., measurement/data collection and reporting/information quality, use of data and 
supports, teacher trust and engagement). 


| In what ways will addressing this 


problem improve systems beyond Into which of the three “big ideas” 
Problem of practice | accountability? | does this problem fit? 


Step 2 


Share the problem statement and your rationale with the group. This can be done by either sharing 
orally or displaying the statements on a wall and allowing everyone to review. 


f f » 
52 “ Copyright © 2018 American Institutes for Research. This handout may be reproduced for district/school use in planning and 
> implementing continuous improvement inquiry cycles. 


Step 3 


As a small group, come to a consensus on the biggest area of difficulty, considering the following 
factors (we suggest rating the PoP statements to help come to a consensus, but this also can just 


be discussed): 


> Importance. On a scale of 1-10, with 1 (least important) and 10 (most important), how important 
is this issue? How much would it matter if we successfully addressed it? Would it be likely to 
lead to either widespread or significant improvements in teacher practice? To what extent is it 
connected to or contributing to other issues that affect teacher practice? 


> Perception/Buy-in. On a scale of 1-10, with 1 (weakest perception) and 10 (strongest perception), 
how strongly do you think others perceive this as a problem? To what extent will others in the 
school support this as a real problem? 


> Feasibility. On a scale of 1-10, with 1 (least feasible) and 10 (most feasible), how feasible might 
it be to address this issue? Can you imagine multiple different strategies that could be tested 
(perhaps on a small scale) in a relatively short period of time (e.g., one semester)? Would tackling 
this issue likely necessitate changes that may be extremely costly, time-consuming, politically 
challenging, or otherwise difficult to implement? 


Step 4 


Draft a problem statement that reflects the group’s thinking regarding the biggest area of difficulty. This 
can be a problem statement shared by a group member, a combination of two or three statements 
shared by group members, or a completely new statement that reflects the group’s thinking and 
discussion. A good problem statement is specific and relatively narrow. Write the problem statement 
in the following box. 


Examples 


Not so good. Teachers do not engage in standards-based planning. 
> A bit broad—would need to define standards-based planning to really determine how 
to address this challenge; root-cause analysis might point in many different directions. 
Better. Teachers are not consistently providing rigorous learning targets and performance scales. 


> Gets at a specific aspect of planning, which could help focus the root-cause analysis and the 
development of a strategy. 
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HANDOUT 5. | Identifying Root Causes 


Discuss the root causes behind the PoP Brainstorm potential root causes—that is, the reasons for 
the problem. These root causes will become the specific challenges to address in this project. 


Step 1 


Using sticky notes, write an explanation for the PoP Keep asking why until you seem to have really 
gotten to the root issue. Focus on system challenges, not symptoms. 


Example Problem of Practice: The highest proportion of teachers with less than “Effective” ratings 
was in the questioning techniques element. We observe this finding for the following reasons: 

> Teachers are not accustomed to asking open-ended questions. 

> Curriculum or instructional materials do not identify opportunities for questioning. 

> Teachers need more examples of how to implement questioning techniques. 

> Teachers need more support on incorporating questioning techniques in their instruction. 

> There are not enough observations to capture this element accurately. 


> The rubric is not clear about what this should look like in practice, and, therefore, evaluators 
rate it inconsistently. 


> Teachers worry that questioning will lead them off topic, thus resulting in not covering all 
necessary material. 


Step 2 


After you have listed all the potential root causes, group these causes into the “big ideas” categories. 
As you group them, you might discover that several root cause statements can be revised as two or 
three statements, resulting in a stronger and more complete statement that identifies the true root 
cause of the problem. 
Categories: 
> Measures and quality of information 
e There are not enough observations to capture this element accurately. 


e The rubric is not clear about what this should look like in practice, and, therefore, 
evaluators rate it inconsistently. 


> Supports and use of data 
e Teachers are not accustomed to asking open-ended questions. 
© Curriculum or instructional materials do not identify opportunities for questioning. 


e Teachers need more examples of how to implement questioning techniques. 
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e Teachers need more support on incorporating questioning techniques in their instruction. 
e Teacher trust and engagement. 


e Teachers worry that questioning will lead them off topic, thus resulting in not covering all 
necessary material. 


Step 3 


As a group, select one root cause to focus on. We suggest rating the root causes to help come to 
a consensus, but this also can just be discussed using the following rating criteria: 


> Importance. On a scale of 1-10, with 1 (least important) and 10 (most important), how important 
is this issue? How much would it matter if we successfully addressed it? Would it be meaningful 
improvement to the PoP? 


> Perception/Buy-in. On a scale of 1-10, with 1 (weakest perception) and 10 (strongest perception), 
how strongly do others perceive this to be an issue? To what extent will others in the district 
believe in or support this as a real cause of the PoP? 


> Feasibility. On a scale of 1-10, with 1 (least feasible) and 10 (most feasible), how feasible might 
it be to address this issue? Can you imagine multiple different strategies that could be tested 
(perhaps on a small scale) in a relatively short period of time (e.g., one semester)? Would 
tackling this issue likely necessitate changes that may be extremely costly, time-consuming, 
politically challenging, or otherwise difficult to implement? 
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HANDOUT 6. | Determine a Strategy and Develop a Theory of Action 


Step 1 


Using a chart similar to the one on the next page, brainstorm a list of possible strategies to address 
the driver/root cause that your group has identified. These are your change ideas. Strategies may 
include approaches or methods that you are already using or new ways to enhance existing strategies. 


Step 2 


Discuss each strategy using the following rating criteria. The row at the top of the table provides an 
example. We suggest rating them to help come to a consensus, but this also can just be discussed. 
Use additional sheets as needed. 


> Likelihood for Success. On a scale of 1-10, with 1 (/east likelihood) and 10 (greatest likelihood), 
how likely do you think this strategy will have a positive impact on the identified problem? 
If implemented well, do you think this strategy can lead to either widespread or significant 
improvements in teacher practice or student learning? To what extent is it connected to other 
issues that affect teacher practice? 


> Feasibility. On a scale of 1-10, with 1 (least feasible) and 10 (most feasible), how feasible 
is it to carry out this strategy? Does this strategy necessitate significant financial, human, or 
other resources that may not be available? Is it likely to be politically challenging or otherwise 
difficult to implement? 


> Scalability. On a scale of 1-10, with 1 (least scalable) and 10 (most scalable), how easy might 
it be to expand this strategy (if successful) to additional teachers or schools? 
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Example 


Likelihood 
Selected root cause Strategy forsuccess Feasibility Scalability Total 
Example 1. Professional development or coaching in questioning 
: ri 8 7 22 
techniques. 


Example 1. Teachers are not accustomed 


to asking open-ended questions. Example 2. Develop materials or extensions to existing materials 


to build in open-ended questions that teachers can integrate 3 7 7 Le 
into instruction. 
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Step 3 


Explain your strategy as a theory of action. How will this strategy/change idea work? Use a simple 
graphic similar to the following to help think about how this strategy will lead to the overall goal 
of an evaluation system that supports teacher practice. 


Now Short-term changes Long-term changes 


Then teachers will do Y ... And the outcome will be Z. 


If we implement strategy X ... 


This diagram depicts your team’s working theory of action for the strategy. Using the diagram, discuss 
the following questions to decide whether changes are required: 


i. 


2. 


If we provide professional 
development or coaching 
in questioning 
techniques, 


do differently? 


e Teachers will improve 
their knowledge of 
questioning and their 
skill in doing it. 


e Teachers will value 
questioning more. 


e Teachers will feel 
more supported. 


e Teacher practice will 
improve, as measured 
by the relevant 
observational 
framework element. 


The district will 
systematically invest in 
additional professional 
development or 
coaching support. 


Why are we prioritizing this strategy? Does it respond to the problem that we identified? 


What, specifically, are we asking all the actors involved in this theory of action to do or 


. What makes us think that if we implement the strategy proposed, it will ultimately improve 


teacher practice and student learning? How will the strategy work to achieve the outcomes 


we expect to see? 
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HANDOUT 7. | Create a Plan for Implementation 


Fidelity of implementation for the strategy your district decides to pursue for this project will be extremely 
important to its success. To understand and measure the extent to which the thoughtful and innovative 
strategies your team implements are actually having an impact on intended outcomes, the strategies 
need to be implemented on time, with care, and according to the plans your team outlines at the start 
of the project. 


This tool provides district teams with a template that they can use to outline plans for implementation, 
including who will carry out which aspects of the strategy, where and when the strategy will be carried 
out, what supports or resources will be needed, and what implementation challenges are anticipated. 
The tool also provides a set of important discussion questions regarding implementation that district 
teams should review and discuss together before completing the implementation plan template. 


Implementation Discussion Questions 


Before your team begins the action-planning phase for how to implement the project strategies, 
consider the following discussion questions as a team: 


1. Is there an existing strategic plan that includes a timeline and process for revisions in this 
area? If so, how can it be used or leveraged for this initiative? 


2. What policies, programs, or processes in this area have been modified recently? If so, what 
implications do these have for this work? Do we expect any district- or state-level changes in 
this area while we are trying to implement these strategies? 


3. Will implementing these strategies impact or conflict with any other school or district priorities? 


4. If we see positive change from implementing these strategies, are we willing to have 
a conversation about how to take the work “to scale”? 


5. What partnerships can we leverage or make to implement these strategies? 


Implementation Plan Template 


Take a few minutes to review the example as a reference. Then use the table that follows to consider 
the important action steps necessary for implementing the project strategies. 


Copyright © 2018 American Institutes for Research. This handout may be reproduced for district/school use in planning and 
implementing continuous improvement inquiry cycles. 


59 


Example 


Implementation plan questions 


| Description 


Example Strategy: Our identified PoP is that teachers need more supports to successfully implement the aspects of 
the instructional practices framework that deal with deepening student content knowledge and skill. The strategy we 
will test during this cycle of inquiry is an in-depth unit- and lesson-planning process in which teachers, coaches, and 
district staff will work together to develop specific lesson plans and strategies to be used during the second semester, 
if possible. Because our district's focus is on literacy and we are especially concerned about our middle schools, we will 


focus our strategy on those schools and grades. 


Where and with whom will this strategy be implemented? 
Be specific. 


Example: We will try this strategy for reading instruction in 
Red, Blue, and Green middle schools with all Grades 6-8 
teachers in those schools. 


Who will be the primary person responsible for 
implementing this strategy? Who will be the primary 
person held accountable for implementing this strategy? 


Example: Mary Smith, an instructional coach, will take the 
lead on implementing this strategy. She will be accountable 
to John Jones, the project coordinator for the district. 


Which team members will be supporting the 
implementation of this strategy, including supporting 
the primary person responsible for implementation? 


Example: Jenny Davis at the district curriculum office will 
work with Mary to help develop the plans, and the three 
principals of the schools will support their teachers’ 
participation with some release time. 


Who needs to be consulted about the implementation 
of this strategy along the way? Who else needs to 
be informed? 


Example: Principals should probably tell other teachers in 
the school who do not teach literacy that some teachers 
are participating in a special project. 


What are the key action steps for implementing this 
strategy (identify up to five)? 


Example: 


Step 1. Mary Smith will organize an initial meeting of 
relevant teachers and staff after school next week. The 
goal of the meeting will be to decide on five units to be 
developed, create a template for them, create a rubric 
to assess their quality/ability to incorporate desired 
elements, and develop a plan for pairs or groups of 
teachers to create unit lessons. 


Step 2. Complete drafts of five lessons. 
Step 3. Meet to review the drafts using rubric. 


Step 4. Revise and finalize the five lessons. 


What is the proposed timeline for implementing 
these action steps? 


Describe an early win for the implementation 
of this strategy. 


Example: 
Step 1. Initial meeting—next week. 
Step 2. Complete drafts of five units—by October 31. 


Step 3. Meet to review the drafts using rubric—by 
November 15. 


Step 4. Revise and finalize five lessons—by December 5. 


Example: Agreement on rubric for quality. 
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Implementation plan questions Description 

What do you expect will be one or two challenges to the Example: People not completing work on time or not 
implementation of this strategy? How will your team attending meetings. We will confirm attendance and 
address these challenges? set up milestones to check on progress. 


What supports or resources are essential for implementing Example: Teachers will need 20 hours to do this work 
this strategy? outside regular time (stipend funds needed). 


Implementation plan questions | Description 


Strategy: 


Where and with whom will this strategy be implemented? 
Be specific. 


Who will be the primary person responsible for 
implementing this strategy? Who will be the primary 
person held accountable for implementing this strategy? 


Which team members will be supporting the implementation 
of this strategy, including supporting the primary person 
responsible for implementation? 


* This example is derived from Bryk et al. (2015). 
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Implementation plan questions | Description 


Who needs to be consulted about the implementation 
of this strategy along the way? Who else needs to 
be informed? 


What are the key action steps for implementing this 
strategy (identify up to five)? 


What is the proposed timeline for implementing these 
action steps? 


Describe an early win for the implementation 
of this strategy. 


What do you expect will be one or two challenges 
to the implementation of this strategy? How will your 
team address these challenges? 


What supports or resources are essential for implementing 
this strategy? 
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HANDOUT 8. | Measures and Predictions 


Practical Measurement Overview 


You cannot improve what you cannot measure. Continuous improvement research involves iterative 
cycles of improvement that focus on a specific PoP A key element of these cycles is practical 
measurement, which enables participants to track implementation and measure improvement. 


What are the elements of a good practical measure? 
> Tracks improvement on the desired outcome. 
> Tracks improvement on the drivers or mechanisms of the desired outcome. 
> Provides guidance for subsequent improvement efforts, so we know how to proceed. 


> Is easy to implement. 
Example* 


In 2010, the Carnegie Foundation for the Advancement of Teaching, in partnership with community 
colleges, established the Community College Pathways Networked Improvement Community. Participants 
aimed to improve community college completion rates and homed in on developmental mathematics 

education as a primary barrier to completion. 


Participants set the following aim: to increase from 5% to 50% the number of students who achieve 
college mathematics credit within 1 year. They hypothesized that providing students with the necessary 
support to put forth effort in the face of challenges would lead to attaining developmental mathematics 
credit. They further hypothesized that a focus on “starting strong” could help students attain the skills 
they need to effectively navigate the challenges. They developed an intervention that provided a package 
of supports. 


To track progress toward their aim, they developed an outcome measure: the percentage of students 
assigned to developmental mathematics courses who enroll in the program and pass both semesters. 
They also developed a primary driver measure to provide formative feedback toward achieving that aim: 
changes in students’ attitudes toward mathematics after 3 weeks. They administered a short survey to 
students that assessed their interest in mathematics, their growth mind-set about mathematics, their 
anxiety about mathematics, and their sense of belonging. 


Each team will develop a plan to monitor progress based on its particular strategies. Use the following 
template to develop this plan and the associated measures. A sample is provided for reference. 


* This example is derived from Bryk et al. (2015). 
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Example strategy: The strategy we will test during this cycle of inquiry is an in-depth unit- and lesson- 
planning process in which teachers, coaches, and district staff will work together to develop specific 
lesson plans and strategies to be used during the second semester, if possible. Because our district’s 
focus is on literacy and we are especially concerned about our middle schools, we will focus our 
strategy on those schools and grades. 


Expected outcomes Measures and data collection Prediction 
Create five new high-quality Count the number of new unit The team will produce five unit 
unit plans. plans created and rubric scores, plans as planned. 
which will be measured by 
reviewing documentation. 
Increase participants’ Ask participants to complete a All participants will report 
knowledge, skill, and value short survey that self-rates their increases in their knowledge 
of questioning techniques. knowledge, skill, and perception of questioning techniques. 
of the value of questioning, 
which will be administered 
during the initial meeting and 
at the end of the process. 
Strategy: 
Expected outcomes | Measures and data collection Prediction 
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